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RESEARCH REPORT 

Cognitively Based Assessment of Research and Inquiry Skills: 
Defining a Key Practice in the English Language Arts 

Jesse R. Sparks & Paul Deane 

Educational Testing Service, Princeton, NJ 


Current educational standards call for students to engage in the skills of research and inquiry, with a focus on gathering evidence from 
multiple information sources, evaluating the credibility of those sources, and writing an integrated synthesis that cites evidence from 
those sources. Opportunities to build strong research skills are critical, yet empirical research demonstrates that students from Grades 
K-16 struggle with inquiry tasks, particularly in online environments. There is a need to create models that will support teachers in 
developing students’ research skills and can be used to develop reliable and valid assessments of such skills while aligning with standards. 
Under the CBAL™ research initiative, we have developed a model of conducting research and inquiry as a key literacy practice in 
the English language arts (ELA). In this paper, we draw on literature from the cognitive and learning sciences — including work in 
discourse processing, science education, educational technology, and information literacy—to provide the theoretical background for 
this key practice. We identify a set of activities and skills that are critical for participating in research; each skill is accompanied by a 
set of provisional learning progressions, which outlines tentative predictions about the qualitative changes in a skill that develop over 
time with appropriate instruction. These learning progressions and their relation to the key practice can be leveraged in the design of 
cognitively based assessments of research and inquiry that are sensitive to students’ developmental level. We conclude, with an example 
design for such an assessment to illustrate how key practices and learning progressions can be integrated to support measurement of 
research and inquiry skills. 

Keywords English language arts; learning from text; reading and writing; multiple-source comprehension; assessment 
doi: 10.1002/ets2.12082 


The Common Core State Standards for English Language Arts and Literacy (National Governors Association Center 
for Best Practices & Council of Chief State School Officers, 2010) calls for students to engage in the skills of research and 
inquiry, with a focus on gathering evidence from multiple information sources, evaluating the credibility of those sources, 
and writing an integrated synthesis that appropriately cites evidence from those sources. In order to achieve the vision 
of college readiness outlined by the standards, students need opportunities to build strong research skills, yet reviews of 
empirical studies have demonstrated that students both in college and grades K-12 experience difficulty with conducting 
research, particularly in online environments (Kuiper, Volman, & Terwel, 2005; Walraven, Brand-Gruwel, & Boshuizen, 
2008). Efforts to support improvements in the teaching and learning of research skills in K- 12 settings are therefore 
critical for helping students achieve true readiness to participate in college and careers where technologically supported 
knowledge work is increasingly the norm (Levy & Murnane, 2004). further, developing students’ proficiency with research 
skills is also likely to benefit their decision-making and problem-solving skills in everyday personal information-seeking 
and social contexts (Brand-Gruwel & Stadtler, 2011; Kovach & Rosenstiel, 2010), underscoring the importance of such 
21st century research skills (Partnership for 21st Century Skills, 2009) for success in work and life. 

Thus, the challenges for K-12 instruction and assessment are to create models that will support English language arts 
(ELA) teachers in developing their students’ research skills (that is, their ability to select, evaluate, and build knowledge 
from multiple sources in response to an inquiry question or problem) and to develop assessments of such skills that provide 
evidence of students’ proficiency. Given the centrality of research skills for college readiness, it is important to develop 
assessments of those skills that effectively measure students’ proficiency while aligning with standards. The development of 
cognitively based assessments of reading and writing literacy under the CBAL™ research initiative at Educational Testing 
Service (ETS; Bennett, 2010; Bennett & Gitomer, 2009; Deane, 2011; Sabatini, O’Reilly, & Deane, 2013) represents an 
important contribution to such efforts. 
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In this paper, we apply the framework of key practices outlined by Deane et al. (2015) to develop a model of profi¬ 
ciency with activities required to conduct research and inquiry in the context of ELA instruction. According to Deane 
et al. (2015), the key practice, conducting research and inquiry, involves “mastery of skills and strategies needed to par¬ 
ticipate in a research community, including the abilities to gather, evaluate, and synthesize information from multiple 
sources, to conduct inquiry and experimentation, and to present information one has learned from sources in appro¬ 
priate forms and formats” (p. 12). Here, consistent with principles of CBAL (Bennett, 2010; Bennett & Gitomer, 2009), 
we draw on literature from the cognitive and learning sciences — including work in discourse processing, science educa¬ 
tion, educational technology, and information literacy fields — to provide the theoretical background for this key practice. 
We present a detailed analysis of the theoretical and empirical work that informed this model, including identification 
of skills that are critical for participating in research as a literacy practice. These key skills are accompanied by a set of 
hypothesized learning progressions — specifying the qualitative changes in a skill that develop over time with appropriate 
instruction—that outline tentative predictions about the development of research skills. While these learning progres¬ 
sions represent our best current understanding of the development of research skills from isolated activities to integrated, 
expert-like performances, it is important to note that they are still provisional and subject to revision based on additional 
empirical evidence and theorization. Finally, we describe ongoing efforts to validate this theoretical model and to use it 
to support assessment design. 

Conducting Research and Inquiry: Framework for a Key Practice 

As described in detail by Deane et al. (2015), the definition of a key practice in terms of an activity system (cf. Brown, 
Collins, & Duguid, 1989; Engestrom, Miettinen, & Punamaki, 1999; Vygotsky, 1978) — coordinated sets of goal-driven 
behaviors within a structure of social norms and expectations for participation—provides a schematic representation of 
the activities that define successful performance within the constraints of that key practice. This schema should be specified 
at a level such that it generalizes across a variety of specific practices and identifies the cognitive foundations as well as 
the enduring understandings and strategies that characterize expert performance of the practice (Zane 2009a, 2009b). In 
the context of activity theory (Jonassen & Rohrer-Murphy, 1999), one must consider the subject (who is performing the 
activity), the object (the ultimate goal or product one aims to achieve), tools (anything the subject uses to achieve the 
goal), and activities (goal-directed actions or sequences of actions that are deployed in an effort to achieve the object) as 
elements that define an activity system. Applying this theoretical framework to an analysis of the research domain, we can 
derive a descriptive model of research and inquiry as a situated practice. 

The practice of conducting research and inquiry—and specifically, participating in a research community to which one 
makes active contributions — requires interactions among multiple people who are engaged in the process of conducting 
investigations and represent the subjects of this activity system. The subjects for this activity system would certainly include 
professional research scientists working in academic or applied settings, but they could also include many others engaged 
in research activities, such as employees, data analysts, undergraduates, and importantly for the current work, K- 12 stu¬ 
dents in ELA classrooms or in disciplinary classrooms where literate research activities are part of the curriculum, as in 
science or social studies domains. The object of research and inquiry is to build a synthetic, deep understanding of a topic 
or phenomenon by reading and integrating information from multiple, reliable sources of evidence and to communicate 
that understanding to others in a coherent way, typically in the form of an explanation or argument, to enter one’s claims 
into an emerging knowledge base and thereby submit them to the scrutiny of one’s peers and other scholars in the field. 
Defining the object of conducting research in this way highlights the importance of conceptualizing research as a social 
practice, involving interactions and exchange among multiple parties — including those with perspectives that differ con¬ 
siderably from one’s own—as a fundamental element of the research enterprise. The tools of research and inquiry are 
varied, depending considerably on the nature of one’s research questions and the methodological approaches one uses to 
find evidence bearing on those questions; these tools could include systematic observation, controlled experimentation, 
or close reading, analysis, and critical evaluation of multiple text or multimedia sources obtained via Internet searches. 
Here, and throughout this paper, we use the term sources to refer to any number of different types of texts, documents, 
or other materials in which information can be transmitted; this includes but is not limited to books, textbooks, newspa¬ 
per articles, journal articles, websites, personal blogs, letters, online discussion forums, political cartoons, text messages, 
tweets, datasets, simulation results, infographics, YouTube videos, or conversations (face-to-face or virtual) with experts or 
interested parties. Importantly, one’s choice of methodological approach and information-gathering strategies (including 
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the types of sources one seeks out) must be appropriate given the questions one wishes to answer or the phenomena one 
wishes to explain, so that one’s claims and conclusions will be considered valid by other members of the research com¬ 
munity. That is, the choice of tools is constrained by both task goals and disciplinary standards that specify the criteria for 
evaluating the validity of knowledge claims (for example, in scientific fields, application of the scientific method and use 
of appropriate controls to rule out confounds, as well as adherence to norms and expectations of the research report genre; 
Goldman & Bisanz, 2002). In this way, research and inquiry has important connections to notions of disciplinary liter¬ 
acy, which emphasize the unique sets of tools and knowledge that experts bring to bear when creating, communicating, 
and using disciplinary knowledge (T. Shanahan & Shanahan, 2008, 2012). Finally, various activities (such as information 
gathering, analysis and synthesis, and communication of results) serve to coordinate the subjects, objects, and tools in the 
service of accomplishing various goals and subgoals that must be satisfied to achieve successful performance. Although 
some activities remain constant across disciplines (i.e., general reading and comprehension strategies), others are specific 
to the discipline (i.e., data collection). 

When viewed as a literacy practice in ELA, research and inquiry is fundamentally a process of reading, writing, and 
thinking with multiple documents (cf. Perfetti, Rouet, & Britt, 1999; Rouet, Britt, Mason, & Perfetti, 1996). The specific 
tools and activities may vary as a function of specific disciplinary constraints. Extant work on disciplinary literacy that 
seeks to characterize the differences in reading and writing activities among the disciplines, in order to develop ways to 
teach and assess literacy activities within those domains (T. Shanahan & Shanahan, 2012), can inform an understanding of 
the literate activities that are required for conducting research in different disciplinary contexts. For example, Latour and 
Woolgar (1986) characterized the practice of scientific research as one of involving translation (i.e., reading and integrat¬ 
ing) and inscription (i.e., writing and communicating) of a multitude of documents (including data and interpretations 
of data) with particular attention to detailing one’s procedures, explaining phenomena, and persuading colleagues that 
one’s findings are credible by engaging in evidence-based argumentation. 

Consider, in particular, how attention to the author of a text varies in importance across disciplines. Research on histor¬ 
ical inquiry reveals that historians must model the author of a text and use this understanding to interpret text information 
(Wineburg, 1991,1998), whereas scientists may use author information as a heuristic indicating the relevance of the topic 
or the quality of the work, using this information to evaluate whether or not a scientific text is worth reading (C. Shanahan, 
Shanahan, & Misichia, 2011). Analyses of think-aloud protocols during disciplinary reading indicate that scientists and 
mathematicians find consideration of the author “a distraction” that does not assist them in interpreting or comprehend¬ 
ing text contents (C. Shanahan et al., 2011); however, in history, consideration of the author of a text is central to the work 
of the discipline. Importantly, disciplinary literacy extends beyond reading to learn specific content and involves learn¬ 
ing about the various practices used by professionals in the context of their work (Goldman, 2012; C. D. Lee & Spratley, 
2010). Whether in scientific, historical, or literary disciplinary contexts, effectively pursuing answers to an inquiry ques¬ 
tion through research requires the successful deployment of a variety of skills, including critical evaluation, analysis, and 
synthesis of information or evidence (in textual, numerical, visual, or graphical modes) drawn from a variety of sources 
presenting a range of views, evidence, and conclusions about the issue under discussion (Bazerman, 1985; Goldman et al., 
2010; Wineburg, 1991). Some scholars have framed these various academic practices as reasoning from multiple sources 
(Perfetti et al., 1999; Rouet et al., 1996). 

When research and inquiry skills are conceptualized in this way, it follows that these skills include reading activities 
required for successful comprehension and learning from single texts, including activating prior topic knowledge, iden¬ 
tifying main ideas, generating inferences, and building a mental or situation model of the text, which can be extended or 
applied to answer questions or solve problems (Kintsch, 1998; Kintsch & van Dijk, 1978). These skills, in addition to related 
skills such as summary writing, outlining, and communicating with others about informational texts, are represented in 
the CBAL ELA framework under building and sharing knowledge, which describes the key practice of comprehending, 
building, and sharing the knowledge one has learned from informational texts. Because this key practice is described in 
detail elsewhere (see O’Reilly, Deane, & Sabatini, 2015), these activities are not dealt with in detail here except to emphasize 
that the model-building skills of building and sharing knowledge are considered prerequisite skills that support students’ 
performance of the key practice, conducting research and inquiry, which is considered an applied literacy skill in the 
CBAL key practices framework (Deane et al., 2015). 

Our definition of the key practice of conducting research and inquiry involves mastery of the knowledge, skills, 
and strategies needed to participate in a research community, including the abilities to gather, evaluate, and synthesize 
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information from multiple sources, to plan and conduct inquiry and experimentation to answer driving questions or 
solve problems, and to present information one has learned in appropriate forms and formats, adapting information as 
needed given the knowledge, interest, and perspectives of one’s intended audience. In contrast to building and sharing 
knowledge, another prerequisite key practice that emphasizes building a coherent mental model of text content (Deane 
et ah, 2015), research and inquiry, involves as a primary emphasis the evaluation and deep synthesis of multiple sources. 
While the key practice building and sharing knowledge sometimes involves learning from multiple texts, its focus is on 
comprehension and understanding, rather than the critical analysis and synthesis of multiple documents that is a central 
focus of inquiry in ELA. Engaging in research is therefore primarily a process of building knowledge from multiple 
authoritative sources where credible resources are used to extend and advance one’s thinking in a process of continual 
improvement of ideas (cf. Scardamalia & Bereiter, 2006). 

The degree to which an individual will be successful in conducting research is a function of knowledge, skills, the 
nature of the required tasks, and available resources, as well as performance moderators (O’Reilly & Sabatini, 2013) such 
as prior topic knowledge, metacognitive and self-regulation skills, and epistemic beliefs about the nature of knowledge 
and knowing, which have been empirically demonstrated to influence inquiry practices (Barzilai & Zohar, 2012). The 
processes of evaluation, integration, and communication depend heavily on the researcher’s content knowledge as well 
as knowledge of specific disciplinary standards and procedures for producing new knowledge (Goldman & Scardamalia, 
2013). 

Development of proficiency with inquiry practices may require attention to supporting learners in building their 
metacognitive skills and strategies and, in addition, explicit modeling of the various literacy and inquiry activities that 
disciplinary experts perform in the service of their professional research practices (cf. Chinn & Malhotra, 2002). This con¬ 
clusion aligns with perspectives on situated cognition, which suggest that learning occurs through interacting with others 
in a social context resembling the real-life conditions in which the learned skills will be applied (Brown et al., 1989). This 
theory also suggests that expertise develops through continued participation in communities of practice (Lave & Wenger, 
1991), wherein novices interact with and learn from experts in the context of their authentic activities (Rogoff, 1990). 
Consistent with these perspectives, the current key practice is specified at a general level, such that the same model can 
be used to characterize learners’ inquiry skills across multiple levels of proficiency from preliminary (novice) to advanced 
(expert) performance. 

Conducting Research and Inquiry Activity Diagram 

Figure 1 presents a schematic representation of the conducting research and inquiry key practice, as defined above. 
This overview, or activity diagram, identifies the major subgoals and associated activities involved in the key practice 
of conducting research and inquiry from multiple evidentiary sources. The activity diagram corresponds to the criti¬ 
cal knowledge, skills, and activities identified from a review of the cognitive and learning sciences literature and is also 
informed by previous analyses of reading and writing literacy practices conducted under the CBAL initiative (Deane, 
2011; Deane, Fowles, Baldwin, & Persky, 2011; Deane, Sabatini, & O’Reilly, 2013; Deane et al., 2008; O’Reilly & Sheehan, 
2009). This diagram illustrates the key activities that literate people might perform if their purpose is to conduct research 
and inquiry from multiple sources of evidence, in order to advance their understanding of a topic, event, or phenomenon 
and to apply that understanding to answer questions, solve problems, or communicate with others. 

Our analysis suggests that the behaviors that characterize expert participation in research and inquiry can be catego¬ 
rized under three major phases of activity. Each phase addresses different sets of goals, which require a variety of reading, 
writing, and critical thinking skills to be deployed; these skills are represented in Figure 1 by the questions corresponding 
to each phase. We describe each phase below and then present a detailed literature review discussing the key aspects of 
each phase in turn. The following activities are involved in this key practice: 

1 . Inquiry and information gathering. This class of activities corresponds to strategies for planning, monitoring, and 
executing information-gathering or data collection strategies. It includes strategies for assessing information needs, 
formulating and revising research questions, and generating plans for obtaining reliable evidence that both bears on 
those questions and fills gaps in one’s understanding of the topic. Performance on these kinds of activities is perhaps 
most likely to be associated with constructs like information literacy, information and communication technology 
literacy, and scientific thinking skills. 
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What do I already know? 

What do I need to know? 

How am I going to find out what I need to know? 
Does my understanding of the subject fit the facts? 
Are there other ways to conceptualize my subject? 



Inquiry and 

Information Gathering 


Conduct 
Research & Inquiry 

Key Practice 


Communication and 
Presentation of 
Results 



Analysis, Evaluation, 
and Synthesis 


What is the current state of discussion in the field? 
What ideas can I contribute that are new? 

Given this focus, how should I present the 
information I have gathered? 

Which parts of it should I include? 

How should I embed those parts in my text? 

How should I document this information? 


Are my sources relevant and useful for my task? 

Are my sources reliable and credible? 

Do my sources agree? 

Can I account for differences between my sources? 
How do their perspectives differ? 

How should I organize the information they provide? 


Figure 1 Framework for the key practice conducting research and inquiry. 


2. Analysis, evaluation, and synthesis. This class of activities corresponds to strategies for comprehending, evaluat¬ 
ing, and consolidating information drawn from multiple information resources. The skills required for successful 
multiple-document comprehension (e.g., Goldman et al., 2011) are best represented in this set of activities, including 
skills required to comparatively evaluate sources that represent different perspectives or are presented in different 
formats and to build a coherent understanding of a topic or event from multiple, possibly conflicting sources. Per¬ 
formance on these activities is likely to depend on the goals of the task, as well as an individual’s topic knowledge, 
understanding of the task, and epistemic beliefs. 

3. Communication and presentation of results. This class of activities comprises the writing skills needed to organize 
and present information drawn from multiple sources, particularly informational texts that describe the results of 
research studies or reviews of research results, and then to present this information to an audience while avoiding 
plagiarism. Communicating and presenting the results of one’s inquiries also requires, at the highest levels, con¬ 
sideration of disciplinary expectations regarding how one should represent, communicate, and evaluate evidence 
obtained from various sources, including norms for engaging in argumentation. 

In Figure 1, each of these three activity types includes a set of questions designed to illustrate the kinds of goals or subgoals 
that individuals might try to achieve in the service of those activities. 

We have represented these activities in a circle with an implied clockwise flow from inquiry and information gathering, 
to analysis and synthesis, and finally to communicating and presenting results. However, it is important to note that these 
activities may occur in various sequences and configurations, as is consistent with the nonlinear and iterative approaches 
that characterize experts’ inquiry practices (Rouet & Britt, 2011). Although generating guiding questions is an important 
early step in the inquiry process, we do not mean to suggest that all inquiries begin with the explicit formulation of 
a question; for example, one might read a text that contradicts or challenges one’s prior understanding, or one might 
encounter conflicting information within or between sources, thereby generating a need to resolve the apparent conflict by 
gathering additional information or data. Individuals might also vary in their engagement in some activities, depending 
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in part on their understanding of what is required to complete the task. A two-step clockwise sequence from inquiry 
and information gathering to communicating and presenting results might represent a simplistic, uncritical approach to 
conducting inquiries in which an individual queries a search engine, quickly scans the first available result, and submits a 
pasted excerpt from the text as one’s response, which is characteristic of young students who view research as a process of 
looking up answers (e.g., Bilal, 2001). In contrast, proficient research and inquiry will include multiple cycles of scanning, 
reading, evaluating, and synthesizing across information sources and repeated cycles of writing, revising, and editing to 
avoid plagiarism or strengthen one’s arguments. In sum, this diagram represents the major activities and activity sequences 
that commonly occur in the practice of conducting research and inquiry, but it does not define a strictly ordered process 
model. 

In the following literature review, we discuss each activity in Figure 1 in terms of its supporting set of skills, considering 
what is known about how these skills develop and, in particular, how students’ performance of these activities differs from 
the performance of experts. This analysis is intended to motivate the conceptual framework and learning progressions that 
are presented in the final section of this paper. The framework provides a model of how research and inquiry skills may 
develop qualitatively through instruction and experience. To the extent possible, we have attempted to identify critical 
qualitative shifts in inquiry skills as well as specific strategies or techniques that may support students in achieving more 
sophisticated levels of performance. The progressions are intended to be used to support assessment design; one of our 
ongoing projects is to develop a series of assessment tasks that are intended to measure different levels of targeted research 
and inquiry skills. 


Literature Review 

Here, we review a number of topics that are central to understanding students’ efforts to conduct research and inquiry. The 
subsections are organized with respect to the major activities in each phase of the practice (see Figure 1). First, we review 
literature relevant to the first phase, inquiry and information gathering, including discussion of skills related to scientific 
and information-literate inquiry, with a focus on research conducted in online contexts. Next, we discuss prior work that 
informs the phase of analysis, evaluation, and synthesis, including work on comprehension and learning from multiple 
documents. In the third phase, communicating and presenting results, we draw on research from cognitive science and 
linguistics to develop a model of writing and citing information from sources. Finally, we review related research on the 
metacognitive and self-regulation skills that are essential to conducting inquiry and recruited across all phases of the key 
practice. Taken together, these lines of research form the evidentiary bases for the conceptual framework presented in the 
final section of this report. 

Inquiry and Information Gathering 

The first phase, inquiry and information gathering, involves skills related to scientific inquiry, such as question generation, 
data collection, interpretation, and hypothesis testing, as well as the skills required to locate relevant information from 
existing data sources, such as content available from the Web. Here, we describe literature that informed our conceptual¬ 
ization of the skills within this phase of the key practice. 

Inquiry as a Key Practice in Education 

The importance of inquiry in education has been evident since John Dewey articulated a belief that learning is “primarily 
an activity which arises from the personal experience of grappling with a problem” (Soltis, 2002, p. 579). Dewey envisioned 
learners coming into the classroom, posing questions deeply rooted in their everyday experiences, and engaging in inquiry 
and debate as they tested their hypotheses about the world. This vision of science as a way of knowing and learning about 
everyday phenomena is underscored by the National Research Council (NRC; 1996), the American Association for the 
Advancement of Science (1993), and more recently, the Next Generation Science Standards (Next Generation Science 
Standards Lead States, 2013), which specify the knowledge, skills, and abilities that constitute proficiency in science. The 
NRC Framework for K-12 Science Education (NRC, 2012) specifies a set of eight science and engineering practices that are 
considered essential for students to learn: asking questions and defining problems; developing and using models; plan¬ 
ning and carrying out investigations; analyzing and interpreting data; using mathematics and computational thinking; 
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constructing explanations and designing solutions; engaging in argument from evidence; and obtaining, evaluating, and 
communicating information. The components of the inquiry process are explicitly treated as practices in order to under¬ 
score that “engaging in scientific investigation requires not only skill but also knowledge that is specific to each practice” 
(NRC, 2012, p. 30). These inquiry practices occur not in isolation but in sequential and intersecting ways. Notably, stu¬ 
dents are expected to engage in these inquiry processes at all grade levels but in successively more sophisticated ways as 
their knowledge and skills develop. 

Inquiry as a practice is also embodied in the National Council for the Social Studies (NCSS) College, Career, and Civic 
Life Framework for Social Studies State Standards (NCSS, 2013), which emphasized the development of deep knowledge 
of social studies topics through posing and answering compelling, deep questions about topics in history, geography, 
civics, and economics. The C3 Framework organizes the structure of each of these disciplines in terms of an inquiry arc, 
which includes four dimensions: developing questions and planning inquiries, applying disciplinary concepts and tools, 
evaluating sources and using evidence, and communicating conclusions and taking informed action. The view of social 
science as inquiry put forward by these standards is consistent with the emphasis on inquiry in both science and ELA 
standards. While in this section we primarily focus on research from scientific and online inquiry contexts, a great deal of 
research on multiple document integration and synthesis (as reviewed in the next major section of this paper) has been 
conducted in the context of history learning (e.g., Britt & Aglinskas, 2002; Wiley & Voss, 1999; Wolfe & Goldman, 2005). 

Fundamentally, inquiry tasks require individuals to coordinate their previous ideas with new evidence gathered about 
a topic or phenomenon (Kuhn, 1989). This coordination process includes at least four components specified by Kuhn and 
Dean (2008): (a) intent, identifying the driving questions that will guide one’s inquiry, and knowing how to productively 
constrain those questions to test a hypothesis — for example, by examining the effect of only one variable at a time; (b) 
analysis, the processes of designing a method for investigating one’s intended question and interpreting the data resulting 
from an investigation; (c) inference, the process of drawing conclusions, or synthesizing information from multiple sources 
(e.g., coordinating theory and evidence); and (d) argument, the process of making and defending a claim to an audience, 
such that one’s claims enter the scientific discourse. Here, we focus primarily on skills related to the intent component, 
including formulating questions, gathering data (i.e., locating relevant information sources), and generating and testing 
hypotheses. 

Authentic Inquiry Versus Inquiry-Based Learning 

Research in science education has considered authentic inquiry—as performed by professional scientists and 
researchers — as distinct from inquiry learning, consisting of classroom-based or researcher-developed tasks that 
engage learners in some facet(s) of the inquiry process (Chinn & Malhotra, 2002). Authentic scientific inquiry requires 
experts to engage in the cognitive processes of generating research questions, designing studies or plans for data collec¬ 
tion, making observations, explaining and critiquing results, developing theories, comparing and coordinating results 
from multiple studies, and studying research reports produced by other scientists with different perspectives (Chinn & 
Malhotra, 2002). These steps are consistent with but not identical to instructional models of inquiry learning, such as the 
inquiry cycle of White and Frederiksen (1998), a model that includes processes of questioning, predicting, experimenting, 
modeling, and applying information. For example, project-based science incorporates inquiry into classroom activities 
(Krajcik et al., 1998). Such projects include a driving question, which anchors learning of meaningful scientific content in 
the context of a real-world problem; investigation and artifact-creation activities that give students opportunities to learn 
concepts, apply information, and represent what they have learned in different forms and formats; collaboration among 
students, teachers, and community members; and the use of technological tools to support one’s inquiries. Experiments 
demonstrate that students better understand course concepts introduced through problem-based learning as compared 
to lecture methods, with these benefits appearing to be driven by a focus on deep engagement with a problem, rather 
than the social or collaborative elements (Pease & Kuhn, 2011). 

Common to both authentic inquiry and inquiry learning in science are the processes of orientation to the research topic, 
hypothesis generation, experimentation, reaching conclusions, evaluation of acquired knowledge, planning and monitor¬ 
ing inquiry activities, and monitoring whether one has accumulated sufficient knowledge to address the research questions 
or hypotheses (de Jong, 2006). These steps are undertaken both by expert researchers and by novices who are still develop¬ 
ing their inquiry skills, but students often have difficulty with some aspects of the practice. In general, accounts of young 
students’ attempts at scientific inquiry reveal that they have difficulty drawing meaningful conclusions from data collected, 
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carrying out procedures in a systematic and logical way, and designing investigations that represent meaningful tests of 
specific hypotheses (Krajcik et al., 1998). College students also experience difficulties with hypothesis-testing procedures, 
such as confusion between descriptive and causal (relational) questions, trouble distinguishing between hypotheses and 
predicted patterns of results that follow from those hypotheses, and difficulty drawing scientific conclusions from observa¬ 
tional data (Lawson et al., 2000). Thus, difficulties with hypothesis-testing procedures that students experience in middle 
school (when many students begin their experiences with conducting scientific inquiries involving designing and con¬ 
ducting investigations) are likely to persist through adulthood without appropriate instructional intervention. 

Similarities Between Scientific and Information-Literate Inquiry 

The view of scientific inquiry as a process is consistent with K-12 ELA perspectives, which argue that teaching research 
and inquiry, particularly in online contexts, should highlight inquiry processes (Guinee, 2004). Notably, Julien and Barker 
(2009) identified several similarities between scientific inquiry and information-literate information seeking, conducting 
online inquiry from multiple sources with a particular focus on obtaining reliable sources (also known as information 
problem solving [IPS]; Brand-Gruwel, Wopereis, & Vermetten, 2005). Both information problem solving and scientific 
inquiry involve the following: 

• Working to achieve a goal. Information-seeking tasks have a goal of locating reliable information in order to satisfy an 
information need or answer a question, whereas scientific inquiry tasks have a goal of developing evidence-based 
explanations for natural phenomena. In both cases, the goal can be viewed as answering a driving question that 
guides one’s information or data-gathering strategies. 

• Activating prior knowledge. This category includes organizing information that is known and information that one 
needs to find out—an important process for both task types. 

• Planning for gathering information or data collection. Information-seeking tasks require students to plan a search 
strategy for locating relevant documents, including identifying search keywords, appropriate alternatives to those 
keywords (i.e., synonyms or combinations of terms), and possible sources of credible information on the topic. In 
scientific inquiry, students generate hypotheses, which guide the design of one’s observations or data-collection 
strategies. 

• Executing strategies to gather information or collect data. When solving information problems, students need to 
carry out their planned search strategies and iteratively revise those strategies as needed depending on the quality 
of results obtained, whereas in scientific inquiry, one conducts experiments or makes observations in order to obtain 
evidence that either supports or refutes a specified hypothesis. 

• Evaluating information. In information-seeking tasks, students evaluate information resources with respect to com¬ 
mon criteria, such as relevance to one’s goals and the reliability and credibility of sources. In scientific inquiry, 
such evaluation tends to be conducted in the service of constructing an argument, paying attention to relationships 
between claims and evidence, and how evidence either supports or refutes one’s tentative claims about the world. 

• Communicating and presenting results. Both types of activities require students to represent their new knowledge in 
some way and to share the results of their research with others. 

Indeed, some science education researchers consider the above information-seeking activities to constitute a subset of 
or complement to the practice of scientific inquiry (e.g., Windschitl, 2008), a view supported by the NRC’s (2012) Frame¬ 
work for K-12 Science Education, in which obtaining, evaluating, and communicating information is included as one of the 
practices involved in science inquiry. In addition, these steps are largely consistent with the inquiry arc described in the 
C3 Framework for Social Studies education (NCSS, 2013). In line with Julien and Barker’s (2009) analysis, we consider the 
processes of locating information sources and developing and testing hypotheses to be complementary activities, which 
students might engage in differentially depending on the specific goals, requirements, or context of a particular investi¬ 
gation. Importantly, inquiry tasks on scientific topics might require students to engage in both direct experimentation or 
observation and the extraction of relevant evidence from text or multimedia materials, whereas inquiry in a historical 
context might be more reliant on the use of extant documentary evidence, requiring students’ textual analysis and argu¬ 
mentation skills (cf. Perfetti et al., 1999; Rouet et al., 1996; Wiley & Voss, 1999). Argumentation is particularly important 
in the practice of research and inquiry across disciplines: scientific argumentation is a social, collaborative process that is 
critical in science inquiry (Duschl & Osborne, 2002; Latour & Woolgar, 1986) as well as history (Wiley & Voss, 1999). 
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Students’ argumentation skills might be appropriately implicated in both scientific and historical inquiry tasks, given 
that one must defend one’s claims or conclusions using evidentiary support in both disciplines, though the specific disci¬ 
plinary standards for evaluating or incorporating evidence into one’s writing might vary (Britt & Rouet, 2012; Goldman 
et al., 2011; T. Shanahan & Shanahan, 2008). It is worth noting that one might not represent the results of all inquiries in 
the form of an argument, but perhaps as a summary, an explanation of an event or phenomenon, a narrative account, or 
some other product, such as an informational website. It is also clear that developing a driving question, planning and 
carrying out an investigation, drawing conclusions, and using evidence to support one’s conclusions are common features 
across scientific and historical inquiry tasks in addition to general IPS tasks. In the next section, we discuss features of IPS 
tasks that are specific to conducting research in online contexts. 

Inquiry and Information Gathering in Online Contexts 

In addition to the skills associated with scientific and information-literate inquiry described previously, it has been sug¬ 
gested that several unique processes may be specifically recruited when engaging with Internet resources. For example, 
some have argued that the digital information age requires the development of new literacies (Leu, Kinzer, Coiro, Castek, 
& Henry, 2013; New London Group, 1996) for locating and successfully building meaning from the relatively small sub¬ 
set of reliable information sources scattered among the multitude of input available via the Web. These digital literacy 
skills include effectively using search engines and digital databases to locate sources, critically evaluating information as 
a function of its content as well as its source; dealing with multimedia resources — texts of various genres and purposes, 
video, audio, photos, animations, and advertisements; constructing a deep, coherent understanding of those resources; 
and interacting with and synthesizing those resources in order to transform them into one’s own creations and produc¬ 
tions, which can be communicated and shared with others via traditional and emerging methods and technologies (Coiro 
& Dobler, 2007; Coiro & Kennedy, 2011; Gilster, 1997; Goldman et al., 2010; Lawless et al., 2012; Leu et al., 2013). In par¬ 
ticular, these perspectives assert that the nature of what it means to be literate in the 21st century necessarily includes 
these additional skills (Lankshear & Knobel, 2006; Leu et al., 2013), given the ubiquity of using online resources from 
a diverse range of sources to answer questions and solve problems in our everyday lives (Kovach & Rosenstiel, 2010; 
Metzger, Flanagin, & Zwarun, 2003). As literate practices change in response to technological and societal develop¬ 
ments, definitions of the literacy construct must also evolve to reflect the nature of those practices as they unfold in 
the everyday experiences of literate individuals (Coiro, 2003; Lankshear & Knobel, 2006; Leu et al., 2013), consistent 
with situated cognition perspectives (e.g., Brown et al., 1989; Lave, 1988). This evolution includes attention to contem¬ 
porary genres, contexts, and purposes for reading and writing, which, by virtue of its current ubiquity, includes the 
Internet. 

Students’ Difficulties With Conducting Online Inquiry 

Students often lack experience with many of the subtasks involved in using online resources to conduct inquiry research 
projects (Eagleton & Guinee, 2002). Students enter the classroom with little previous experience with the following tasks: 
choosing an appropriate topic, audience, and purpose for the assignment; accessing relevant background knowledge; envi¬ 
sioning or planning for the final product of one’s inquiry; specifying research questions; recognizing useful information; 
gathering and synthesizing information; assembling information into a written product and revising that product; and 
presenting information to an audience. Using the Web for research tasks poses particular challenges for developing stu¬ 
dents (Bilal, 2000, 2001, 2002; Braasch et al., 2009; Henry, 2006; Hirsh, 1999; Hoffman, Wu, Krajcik, & Soloway, 2003; 
Kafai & Bates, 1997; Kuiper, Volman, & Terwel, 2005, 2009; Large & Beheshti, 2000; Lorenzen, 2001; Rouet, 2006; Rouet 
& Coutelet, 2008; Schacter, Chung, & Dorr, 1998; Wallace, Kupperman, Krajcik, & Soloway, 2000). Therefore, students’ 
proficiency with online inquiry skills varies widely (Kiili, Laurinen, & Marttunen, 2008). Students’ difficulties with con¬ 
ducting online inquiry may be due to a lack of familiarity with or strategic knowledge about particular steps in the inquiry 
process (Guinee, 2004), as well as developmental constraints on their reading skills (Lorch, Lorch, Gretter, & Horn, 1987; 
Rouet & Eme, 2002). 

Kuiper et al. (2005) presented a review of literature from information literacy and education perspectives examining 
Web search and information-processing behavior of upper elementary and middle school children. Based on their sum¬ 
mary of 24 empirical studies, the authors characterize students’ interactions as demonstrating the following characteristics: 
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• A preference for using a browsing strategy (defined as “skimming over information and selecting choices”; Borgman, 
1995, p. 666) rather than using directed keyword searching strategies 

• Difficulty in formulating appropriate search keywords to locate information effectively 

• A limited tendency to explore a variety of sources and frequent reliance on familiar or well-known websites 

• Little patience for reading through long lists of search results, and disengagement or boredom when encountering 
large amounts of text 

• Difficulty with reading and comprehending extended texts, due to complexities in text structure and lack of clear 
formatting (e.g., headings and advance organizers) 

• A focus on collecting and accumulating factual knowledge about a topic rather than thinking about how the infor¬ 
mation could be applied to the task 

• A tendency to search for a single correct answer (Bilal, 2001); for example, Wallace et al. (2000) found that sixth 
grade students tended to view an online science inquiry task as one of finding the right answer or finding an ideal 
website rather than one of using evidence gathered from multiple resources to construct an answer to an inquiry 
question 

• A tendency to revise one’s research question when literal answers are not readily obtained, rather than revising one’s 
search strategy to obtain more relevant or useful results 

• Little attention to reading, processing, and analyzing the information found 

• Difficulty in evaluating the relevance of Web information for one’s purposes (Bilal 2000, 2001; Hirsh, 1999; Large 
& Beheshti, 2000); in particular, Kafai and Bates (1997) concluded that as children develop from first through sixth 
grade, they are better able to explain why a website is useful or not, while younger children can only indicate whether 
or not a website is useful 

• Difficulty in attending to and evaluating the reliability of Web information, with little attention to authority, exper¬ 
tise, credibility, and trustworthiness (Hirsh, 1999, Wallace et al., 2000) 

Given these difficulties, Kuiper et al. (2005) concluded that educators can support elementary and middle school stu¬ 
dents in conducting online research in the following ways: help students to develop critical thinking skills, including how 
to assess the relevance and reliability of online information given a particular assignment or research question; help stu¬ 
dents acquire searching skills; provide reading strategy instruction specific to reading extended and discontinuous text 
on the Web; engage students in critical thinking about the visual/multimedia nature of Web texts and their consequences 
for understanding and communicating; engage students in conducting inquiry-based assignments that require search¬ 
ing for information online; and take into account the student and task characteristics that influence students’ choice of 
information search and other online inquiry strategies. 

The Centrality of Reading Skills for Online Inquiry 

The vast amount of information available and the absence of an obvious linear structure on the Web pose particular 
challenges for poor readers (Coiro, 2003) who not only lack patience for wading through lengthy online texts, but also may 
lack the decoding, vocabulary, or comprehension skills necessary to effectively navigate, process, and evaluate information 
from online sources, where it is especially important to question the accuracy of the content and the motives of the author 
(Coiro & Dobler, 2007). Conducting inquiry from online sources, according to Kuiper et al. (2005), is “a question of 
reading critically—reading and simultaneously evaluating the relevance and reliability of what you are reading” (p. 305). 

The importance of selecting appropriate information resources, and being able to read and evaluate those resources 
as a function of their source characteristics, was further emphasized by Goldman et al. (2011), who presented a model 
of multiple-source comprehension developed for the purposes of building assessments of deep comprehension of texts. 
This model was designed to capture the important elements of the domain of multiple-source comprehension specific 
to the context of historical and scientific inquiry tasks. Derived using an evidence-centered design approach (Mislevy, 
Almond, & Lukas, 2003; Pellegrino, Chudowsky, & Glaser, 2001), the model specifies the knowledge and skills that stu¬ 
dents should be able to demonstrate as evidence that they are proficient with some aspect of inquiry from multiple sources 
(i.e., interpreting the inquiry task, searching for and gathering resources, attending to and evaluating documents’ source 
information, interpreting and synthesizing information from sources, and applying information extracted from sources 
to achieve one’s task). The authors emphasized sourcing, searching for, and selecting resources and interpreting and 
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synthesizing resources as activities in the model that are more distinctly characteristic of multiple-text versus single-text 
reading situations (Goldman et al., 2011). We return to this model in the section on analyzing, evaluating, and synthesiz¬ 
ing multiple sources. Next, we focus on a critical aspect of source selection — namely, the ability to identify information 
relevant to a topic. 

Students’ Reliance on Cues to Information Relevance 

To successfully select information resources that will be useful for a particular task requires inquirers to attend to and 
integrate multiple cues to textual relevance (Rouet, 2006). However, as noted by Kuiper et al. (2005), young children’s 
conception of relevance is limited, which might lead to difficulty in subsequent inquiry activities. Consistent with analyses 
of adults’ relevance judgments (Xu & Chen, 2006), Hirsh (1999) observed that fifth-grade students were most reliant on 
the criteria of topicality and personal interest when searching for information in a constrained research task, with a shift 
toward reliance on novelty of information as work on the project progressed and students focused on finding information 
that filled gaps in their expanding knowledge of the topic (cf. Kuhlthau, 1997). However, these students mostly relied on 
the titles or summaries of documents (i.e., abstracts in bibliographic records or summaries from a list of search engine 
results) when making judgments of topicality, and they had difficulty detecting relevance when titles were nonliteral or 
did not explicitly refer to the research topic (see also Kafai & Bates, 1997). Some students overlooked resources that did 
not directly match their research topics, leading Hirsh to characterize them as “concrete thinkers [who] have trouble with 
anything that is not an exact fit with their understanding of the question” (p. 1279). 

Cues to the relevance of information include both surface cues and deep semantic cues, which in the context of a 
search engine results page, could involve bolded search keywords or use of phrases associated with—but not explicitly 
matching—words used to describe the topic (Rouet, Ros, Goumi, Macedo-Rouet, & Dinet, 2011). In a set of experiments, 
Rouet et al. (2011) showed that fifth-, seventh-, and some ninth-grade students are sensitive to surface features of search 
results, being more likely to select website titles that emphasized matching keyword terms but that were irrelevant to the 
topic upon deeper semantic inspection. In contrast, 12th-grade students were able to overlook misleading surface features 
and to select websites purely on the basis of their semantic relevance (i.e., their “aboutness”; Xu & Chen, 2006, p. 962). 
Prereading to build knowledge of the topic prior to selecting resources reduced fifth and seventh graders’ reliance on 
surface cues, but only for good readers who were able to successfully comprehend the prereading text and apply what they 
had learned from that text to constrain their search. Specifically, good readers who engaged in this prereading task were 
less likely to select semantically irrelevant websites with cued surface features (capitalized search keywords in the title), 
and both fifth and seventh graders were able to profit from this intervention. Poor readers will likely require alternative 
supports to develop their skill in selecting sources on the basis of topical relevance, as prereading scaffolds are insufficient 
for helping these students make appropriate choices among alternative information resources. 

Development of Strategies for Locating Relevant Information 

Students can effectively employ strategies for locating relevant information, though their preferred strategy seems to follow 
a developmental trajectory from random browsing and linear reading to top-down, goal-driven use of text organizers, 
headings, and other elements of document formatting (Rouet, 2006). When locating information in texts, third graders 
primarily use a linear reading or browsing strategy; fifth graders prefer use of tables of contents, document headings, and 
topic sentences; and seventh graders tend to use the index to look up the topic of interest (Rouet & Coutelet, 2008). Thus, 
a progression from more linear reading strategies to more top-down, relevance-driven location strategies is apparent as 
students develop metatextual knowledge about how structural features such as headings, tables of contents, and indices 
function in texts (Rouet & Erne, 2002). However, although students can rely on these organizers and headings when 
locating information within books, it is more difficult to apply these strategies to reading Internet texts due to differences 
in the types and consistency of headings and organizational cues across different websites (Large & Beheshti, 2000). Even 
with the development of metatextual knowledge relevant to hypertext environments, limited topic knowledge, including 
knowledge of topically related vocabulary, constrains the ability to identify information as relevant at a deeper semantic 
level (Rouet et al., 2011) if it does not match the topic-related terms the student has in mind. Unfortunately, opaque titles 
are common on the Web, making it easier for children to be misled or to mistakenly classify information as irrelevant 
(Kafai & Bates, 1997). 
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Difficulty locating relevant information can lead students to feel frustrated with the research process (Large & Beheshti, 
2000), particularly when students have something specific in mind but are having difficulty locating that information. 
Instruction in specific search strategies, such as how to combine multiple terms to limit search results or how to use quo¬ 
tation marks to locate verbatim phrases, might be helpful in reducing such frustration (cf. Kuiper et al., 2005). Specifying 
appropriate search keywords is critical for locating relevant and reliable information, but selection of keywords is con¬ 
strained by an earlier step in the inquiry process — specifying research questions to be investigated. Literature related to 
students’ skill in generating research questions is addressed in the next section. 

Asking Driving Research Questions 

Developing driving questions represents a critical first step in science inquiry, and the types of questions that students 
ask can be used as evidence of students’ thinking (Yarden, Brill, & Falk, 2001). Questions such as What do you think 
about this? What would you like to know? and How will you find out? are considered to serve as an initial foundation for 
engaging students in an inquiry (Kuhn & Dean, 2008). However, the questions that students propose to investigate must be 
framed appropriately to support the task of research. Specifically, engaging students in inquiry requires first ensuring that 
students understand inquiry as a process of “seeking information that will bear on a question whose answer they do not 
already know” (Kuhn & Dean, 2005, p. 867). Otherwise, such activities might be misinterpreted as attempting to collect 
evidence to reinforce one’s existing knowledge or to simply look up answers to straightforward, factual questions (Kuhn 
& Dean, 2005, 2008). While obtaining answers to simple factual questions maybe an important component of building 
knowledge about a topic, such questions may not require multiple sources to obtain a reliable answer and are therefore 
not rich enough to support a sustained inquiry. Therefore, students must understand the need to formulate a question as 
well as how to develop effective questions that are appropriate for research. This may include both descriptive questions, 
which can be explored by making systematic observations and comparisons without directly manipulating variables, and 
relational questions, which examine associations between variables or phenomena and employ experimental designs to 
compare and contrast multiple levels of those variables to evaluate the nature of those relationships (Krajcik et al., 1998). 
Further, when students conduct online inquiry with a clear driving question in mind, their searching and reading is more 
meaningful (Burbules & Callister, 2000), as these processes are enacted from the perspective of locating answers to specific 
questions or solutions to certain problems. 

Challenges in Generating and Evaluating Research Questions 

Evidence suggests that middle school students have difficulty formulating appropriate questions and judging the appro¬ 
priateness of others’ questions (Hoffman et al., 2003). Specifically, Krajcik et al. (1998) observed that seventh graders have 
difficulty assessing the scientific value of questions due to a lack of background knowledge, relying instead on personal 
experience, preferences, or uniqueness (i.e., distinctiveness of a question from the ones being investigated by other student 
groups) as criteria for choosing a question to investigate. Students’ questions were often driven by available tools or instru¬ 
mentation (i.e., feasibility and access to equipment) rather than by an analysis of what is known, unknown, and necessary 
to find out about the research topic. Students failed to consider how the evidence being collected would relate to their 
driving question or to particular predictions. Scardamalia and Bereiter (1992) observed that students in fifth and sixth 
grades tend to construct low-level, factual questions rather than questions that can meaningfully extend their knowledge 
of a topic. Students’ feedback on their peers’ questions also tends to be superficial and not particularly constructive (Scar¬ 
damalia & Bereiter, 1992). Building adequate topic knowledge is critical to enable students to ask good questions. Krajcik 
et al. (1998) concluded that inquiry activities should provide sufficient time for students to explore and locate relevant 
background information in order to support them in evaluating the appropriateness of their questions and in generating 
specific hypotheses and predictions from those questions. However, even undergraduates have difficulty distinguishing 
descriptive and relational (causal) questions (Lawson et al., 2000), suggesting that knowledge of the types of questions 
one might ask about a research topic is important for effective inquiry, beyond simply learning more about the topic. 

Supporting Students in Generating Questions 

Because of the difficulties students experience with posing research questions, some scholars suggest that question gener¬ 
ation should be constrained. Kuhn & Dean (2005) argued that using prompts or templates can help students to manipulate 
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or examine a single factor at a time, making their inquiries more manageable and more likely to yield appropriate causal 
inferences in later phases of inquiry. Constraining student questions in productive ways can help them to focus on impor¬ 
tant elements of the research question being investigated, such as relationships among key variables (Kuhn & Pease, 2008). 
However, it is important not to constrain students’ questions too strictly, particularly in online contexts. Self-generated 
questions appear to facilitate better online inquiry performance relative to assigned or imposed research questions (e.g., 
Bilal, 2001, 2002; Dresang, 1999). This finding suggests that it is important for students to generate their own research 
questions, despite the associated challenges, to support students’ engagement with inquiry activities. As noted above, stu¬ 
dents’ questions are constrained by their prior knowledge of the research topic; therefore, encouraging students to access 
and build on their relevant knowledge prior to question generation can support better formation of research questions 
with a clear goal. Evidence also suggests that middle school students can develop and research workable, appropriately 
worded driving questions with appropriate instructional supports, such as making explicit connections between ques¬ 
tion generation and the overall purpose and goals of the inquiry task (Kuiper et al., 2009). Having a deeper sense of the 
purpose of a research question—to inquire about an aspect of the topic for which they lack adequate knowledge or evi¬ 
dence, to enrich their understanding, or to aid in solving a problem — might help students better formulate and revise 
their questions. 

Questioning as a Metacognitive Inquiry Process 

Others have characterized questioning as a central element of the research process (e.g., Burke, 2002; Moore, 1995). 
According to Burke (2002), students must engage in continuous questioning when learning how to “read” online texts, 
including such questions as What do I want to know? For what purpose? and What do I need to find out to answer my 
questions? This attention to purpose, or task-oriented relevance (Rouet & Britt, 2011), might help students avoid getting 
lost or disoriented when using the Web to gather information (cf. Kuiper et al., 2005). In this case, questioning serves a 
metacognitive or self-regulatory function that helps students regulate their behavior throughout the inquiry process. We 
return to a discussion of the metacognitive aspects of conducting inquiry in the final section of the literature review. 

Analysis, Evaluation, and Synthesis of Multiple Sources 

We have conceptualized the practice of research and inquiry as one involving the extended analysis, evaluation, and syn¬ 
thesis of multiple sources of information, including texts and documents. The integration of multiple resources poses 
some distinct challenges in contrast to reading and learning from single texts; we review current research on the skills 
required for multiple-text comprehension below. 

Cognitive Models of Multiple-Document Reading Comprehension 

Scholars have argued that reading and writing from multiple sources requires more complex skills than does single-text 
comprehension (e.g., Goldman, 2004). In the case of multiple sources, the distinct texts are written by different authors, 
often writing for different purposes; texts may contain different levels of cohesion, may contain different text structures, 
and may demand greater or fewer knowledge-based inferences of the reader (e.g., O’Reilly & McNamara, 2007). In addi¬ 
tion, there are often no explicit links and connections across texts, so the reader has to infer them by recruiting prior 
knowledge and making cross-text comparisons (Britt & Aglinskas, 2002; Perfetti et al., 1999; Wineburg, 1991). There¬ 
fore, multiple source understanding may demand different and potentially deeper cognitive processing as compared to 
comprehension of single texts. 

The Documents Model 

Expert performance in comprehending multiple documents is characterized by building a coherent mental representation 
or documents model (Perfetti et al., 1999), which captures the content of the documents and relationships among their 
content (situation model) as well as information about the documents’ sources (e.g., author, publication venue, date, or 
document type information; document nodes), which content was provided by sources (e.g., according to Jones, from 
the New York Times — source-content links), and the relationships among those sources (e.g., agrees with, disagrees with, 
responds to, refutes, reinforces, replicates; source-source links). However, this model characterizes how multiple sources 
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on a topic might be represented in memory (i.e., with important or distinctive content being more likely to be associated 
with or linked to source information). That is, it describes the nature of the product of multiple document comprehension 
rather than the cognitive processes that might be involved. 

The MD-TRACE Model 

In a major extension of the documents model framework, Rouet and Britt (2011) proposed the MD-TRACE model, which 
describes both the processes and products involved in constructing a documents model (Perfetti et al., 1999), particularly 
in the context of a specific task or assignment (e.g., writing a research paper from sources, writing an argument in favor 
of one causal explanation for an event, etc.). This model describes multiple document comprehension in the context of 
inquiry tasks as a process involving five steps. The following steps may be cycled through in an iterative fashion until all 
information needs and task goals are satisfied (Rouet & Britt, 2011): 

1. creating (or updating) a mental model of the task and associated goals and criteria for successful task completion; 

2. assessing information needs, based on the quantity and quality of one’s prior knowledge about the topic, including 
asking questions that need to be answered; 

3. interacting with documents, which involves 

a. locating and selecting relevant information resources, 

b. reading and comprehending the resources, constructing a mental model of their contents, and 

C. creating (or updating) a documents model encoding the relationships among multiple documents with 
respect to content and source-based links; 

4. applying the information learned from sources to create (or update) the required task product or artifact (e.g., an 
argument in support of a particular position); and 

5. evaluating the task product with respect to goals, requirements, and criteria for successful completion. 

The MD-TRACE model treats task-based or goal-based relevance considerations as a central feature of interacting 
with multiple documents, consistent with a relevance framework (cf. McCrudden, Magliano, & Schraw, 2011). Research 
from this perspective has demonstrated that task goals or instructions for reading have an important influence on reading 
comprehension in that such goals help define what information is relevant to attend to (McCrudden, Magliano, & Schraw, 
2010) with consequences for memory and recall (Pichert & Anderson, 1977). Goals for reading also impact the reader’s 
standards for building a coherent understanding of the information (Linderholm & van den Broek, 2002; van den Broek, 
Lorch, Linderholm, & Gustafson, 2001). That is, task goals can influence what readers attend to during reading, which 
influences what they remember from the text, in addition to influencing the amount of effort that students exert in order 
to deeply understand text content, with reading to study leading to greater coherence-building processes than reading for 
entertainment (van den Broek et al., 2001). These relationships suggest that task goals have an important influence on read¬ 
ers’ comprehension and learning from texts. The influence of task is especially important in multiple-text reading tasks. 

Task-oriented reading as described in the MD-TRACE model is exemplified by activities such as reading in order to 
write a synthesis (Spivey & King, 1989) and is consistent with constructs such as online reading comprehension (Coiro 
& Kennedy, 2011), online inquiry (Hoffman et al., 2003; Wallace et al., 2000; Zhang & Quintana, 2012) and information 
problem solving (Brand-Gruwel et al., 2005; Moore, 1995; Raes, Schellens, de Wever, & Vanderhoven, 2012) as used in 
educational and information sciences literatures, where students’ conception of the task they are engaged in, what infor¬ 
mation is needed to address that task, and how to go about obtaining, evaluating, and using that information figure in a 
range of models and frameworks. Despite this iterative reading and evaluation process, many students, including under¬ 
graduates, have a tendency to simplify complex and open-ended inquiry tasks, treating them as a task of seeking a single 
answer to a constrained question and looking for a single source that might straightforwardly reveal the desired answer 
(Wallace et al., 2000; Wiley et al., 2009). Full engagement in multiple-source inquiry tasks involves the use of multiple 
strategies; research on these strategies is described in the next section. 

Strategies That Support Comprehension of Multiple Documents 

Research has attempted to describe strategies that support successful comprehension and integration of multiple sources. 
For example, in a landmark study of historians’ reading of multiple documents, Wineburg (1991) identified three heuristics 
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that expert historians used when making sense of a historical event: sourcing, corroboration, and contextualization. Sourc¬ 
ing involves attending to and evaluating the source of a document prior to reading the content, as well as using the source 
to help interpret the contents of a document, including evaluating authors’ perspectives and biases and their influence 
on the presentation of ideas. Corroboration involves comparing information across two or more sources or compar¬ 
ing information between sources and one’s prior knowledge, including comparing perspectives, identifying consistencies 
or discrepancies between perspectives, and identifying consistencies or discrepancies between the content of different 
documents. Corroboration is thought to be critical to strategic processes that support the creation of an integrated under¬ 
standing of multiple sources (Braten, Britt, Stromso, & Rouet, 2011). For example, Kobayashi (2009) found that when 
undergraduates read a set of two texts that presented conflicting views on a topic, those students who spontaneously wrote 
notes that indicated the relationship between the two sources were more likely to demonstrate improved comprehension 
of the relationships between the texts (indicative of a more elaborated documents model; Perfetti et al., 1999). Further, 
spontaneous construction of notes that summarized the individual texts predicted recall of the arguments contained in 
the texts as well as cross-text comprehension, suggesting that prompting students to construct summaries of texts they 
encounter in their research will facilitate comprehension of relations across multiple, conflicting texts (Kobayashi, 2009). 
Thus, corroboration strategies can contribute to improved comprehension (Braten, Britt, et al., 2011; see Gil, Braten, Vidal- 
Abarca, & Stromso, 2010a, 2010b). Contextualization involves situating a document within the sociohistorical context in 
which it was produced, including publication date. Contextualization is most important for historical inquiry (NCSS, 
2013) and is related to considerations of perspective-taking and judgments of relevance. 

Evaluating the Usefulness of Information 

Discussions of relevance in multiple-document contexts often refers to the degree to which information is perceived to 
be useful for completing a particular task or achieving a goal, or task-based relevance (McCrudden et al., 2011; Rouet 
& Britt, 2011). But what factors comprise judgments of relevance? Evidence from factor analyses of adults’ situation- 
based relevance judgments in online information-seeking tasks (Xu & Chen, 2006) indicates that relevance judgments 
of websites are primarily driven by judgments of topicality (whether the text is about the targeted topic) and novelty 
(whether the text provides novel information about the targeted topic). Reliability (i.e., accuracy, consistency with facts) 
and comprehensibility of the content were also significantly related to judgments of relevance, but to a lesser degree than 
topicality and novelty. These four factors or dimensions correspond to the Gricean maxims (Grice, 1975) of relation (be 
relevant), quantity (be informative but not too informative), quality (do not say things for which you have no evidence), 
and manner (communicate so that you are able to be understood). Topicality and novelty can be thought of together as 
roughly “informativeness,” or the degree to which a source provides new, unique information that is about the topic of 
interest (Xu & Chen, 2006, p. 970). 

However, when thinking about supporting students in developing their multiple-document inquiry skills, this notion 
of relevance must be extended from a conception of relevance as giving new information about a topic to one of task-based 
relevance, where the primary question learners should ask themselves is not Is this information new to me? but rather Is 
this information useful for completing my task goals? This potentially more complex conception of relevance cannot be 
achieved until a student first understands that relevant information must truly be about the topic rather than containing 
words that are associated with or correspond to words used in describing the topic (i.e., keyword overlap). Arguably, 
the conception of relevance as including both topically related and novel content — that is, contributing information that 
fills a knowledge gap or extends one’s understanding—is an intermediate position between mere topical relevance and 
task-based relevance. 

Evaluating the Reliability and Credibility of Information Sources 

Although students have some success in evaluating relevance, evidence suggests that evaluation of the reliability of sources 
is more difficult for them, despite an early developing ability to evaluate information sources. Developmental cognitive 
research reveals that children as young as ages 3-4 can distinguish among reliable and unreliable sources (i.e., adults). 
When children reach age 3, they display a marked preference for trusting and seeking information from knowledgeable 
over ignorant informants; they also begin to recognize particular cues to the speaker’s confidence (Baldwin & Moses, 
2001). For example, in word-learning studies, 3-year-olds who are given a choice between a word-referent link from 
a knowledgeable speaker and one who explicitly states that he or she does not know yet still offers a guess as to the 
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word-referent link will reliably learn the word provided by the knowledgeable speaker (Koenig, Clement, & Harris, 2004; 
Koenig & Harris, 2005a, 2005b). While both 3- and 4-year-olds demonstrate better learning of words from knowledgeable 
as compared to ignorant speakers, 3-year-olds cannot distinguish between speakers’ ignorance and uncertainty (Sabbagh 
& Baldwin, 2001). Various developments in children’s interpersonal abilities or theory of mind (Flavell & Miller, 1998) are 
thought to contribute to the development of the ability to evaluate the reliability of sources as children grow throughout 
the preschool years, including skill in making interpersonal inferences (Baldwin & Moses, 2001). Thus, the prerequisite 
skills for evaluating the sources’ accuracy and knowledge emerges from the development of social processes. 

Evaluating the Credibility of Texts and Documents 

While such findings indicate that children can learn to discriminate among sources in familiar oral contexts at an early 
age, critical attention to the characteristics of sources of text information, particularly in the case of digital documents, is 
unlikely to develop without instruction. Research on middle school students shows that they are unlikely to attend to the 
authority, expertise, or credibility of sources when evaluating their usefulness to a task (Braasch et al., 2009; Hirsh, 1999; 
Hoffman et al., 2003; Large & Beheshti, 2000). Students are not particularly likely to evaluate the accuracy of information 
obtained online (Hoffman et al., 2003), though some students acknowledge that the Internet introduces an element of 
uncertainty about the reliability of information. For example, a sixth-grade girl interviewed by Large and Beheshti (2000) 
remarked, “Anyone can put anything on the web whether it is true or not.... People writing books make mistakes too, but 
generally the idiots who want to have fun screwing people up put it on the web” (p. 1074). This is perhaps a cynical view 
from a sixth grader, but it reflects the idea that one must scrutinize the accuracy and the authors’ motivations for posting 
information online. 

Sixth-grade students might pay some attention to trustworthiness when evaluating sources online, but those evalua¬ 
tions are limited to the deployment of simplistic heuristics like considering the top level domain of a URL (e.g., .gov and 
.edu are trustworthy, while .com and .org are less trustworthy; Hoffman et al., 2003). Students generally overlook source 
features such as accuracy, trustworthiness, bias, and expertise. Notably, even undergraduates tend to overlook informa¬ 
tion bearing on source credibility, unless given explicit tasks or instructions to consider and apply this information to 
one’s reading (Britt & Aglinskas, 2002; Rouet et al., 1996; Sparks & Rapp, 2011). Rather than relying on evidence of the 
sources’ expertise or potential biases, college students often rely on an evaluation of whether the information is plausible 
or consistent with their prior knowledge and give no further scrutiny unless a violation is detected (Sparks, 2013). 

It is clear that in the service of conducting research, students of all ages must not only comprehend, analyze, and evaluate 
text content, but they must also consider content in light of the characteristics of its source in order to appropriately 
build coherent understandings of important concepts by reconciling information from different sources (Braasch, Rouet, 
Vibert & Britt, 2012; Rouet & Britt, 2011). It may be the case that only higher-knowledge students are likely to engage 
in critical evaluation of sources in the absence of any specific prompting to do so (Goldman, Braasch, Wiley, Graesser, & 
Brodowinska, 2012). However, younger students can learn to attend to and evaluate source information with appropriate 
instruction. For example, a recent study found that fourth- and fifth-grade students learned to evaluate sources’ expertise 
(i.e.. Which source knows more about this topic?) with instruction that engaged the students in comparing the arguments 
of two sources that varied in their expertise (i.e., a research scientist or a local resident; Macedo-Rouet, Braasch, Britt, & 
Rouet, 2013). By requiring students to comparatively evaluate the expertise of sources in order to resolve inconsistencies in 
text contents, instruction can support students in development of sourcing skills. Another element of attention to sources 
involves appropriate citation methods; this issue is discussed in the following section describing the final phase in the 
inquiry process, communicating and presenting one’s findings to an audience. 

Communication and Presentation of Results 

In this section, we describe research on writing from multiple sources, including tasks such as synthesizing the results of 
multiple sources and citing those sources while avoiding plagiarism. 

Writing From Multiple Sources 

While models of multiple-document comprehension suggest that “knowledge of the norms and conventions for 
communicating disciplinary content play an important role” (Goldman et al., 2011, p. 180) in how students apply 
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information to their task, these models do not explicitly address how genre or disciplinary conventions for writing 
influence one’s task product or what components of students’ writing skills might be recruited and required by multiple- 
source comprehension tasks, particularly when those tasks culminate in written arguments or explanations that respond 
to inquiry questions. Models from the reading comprehension perspective also fail to specify the types of written or 
multimedia products that students might be expected to create as evidence of their comprehension and integration 
of evidence from multiple sources. Beyond writing an argument essay (e.g., Britt & Rouet, 2012), students might be 
expected to integrate information from multiple sources in research papers, informational brochures, websites, blog 
entries, newspaper articles, critiques of research articles, annotated bibliographies, abstracts, memos, presentations, 
posters, and so on. Comprehension models do not provide much guidance with respect to evaluating students’ skill in 
integrating information from sources with different modalities (e.g., integrating text and images, or text and graphs, or 
data tables); writing well-formed connected text in different genres (e.g., explanation vs. argument tasks); tailoring one’s 
writing for different audiences (e.g., authority figures, community members, teachers, parents, peers, novices, experts, 
etc.), which requires a consideration of their presumed level of knowledge, interest, and perspective on the topic; or in 
incorporating references and appropriate citations to the work of others. 

Citing and Incorporating the Work of Others 

The practice of citation is critical for academic research, which seeks to build knowledge by extending, rethinking, 
and creatively synthesizing one’s own work with the work of others to indicate the important influences on one’s 
thinking and to give credit to the thinking of others (cf. Hyland, 1999). Horn (2001) reviewed several functions of 
citations in academic writing, including supporting the claim or conclusion one wishes to advance, showing alter¬ 
natives or counter-evidence to one’s conclusions, and giving recognition to the cited author(s). Importantly, Horn 
suggested that the use of citations to previous works helps the writer not only to connect one’s ideas to the ideas of 
others but also to build a consensus around the validity and strength of a particular conclusion. For example, by citing 
important and seminal sources in a discipline to support one’s conclusions, the writer attempts to communicate the 
validity of those conclusions from the perspective of that discipline or of a particular theoretical frame. As Latour 
and Woolgar (1986) put it, there is a credibility-building function of citations in that the objective of citations is to 
“persuade colleagues that they should . . . accept and borrow this assertion as an established matter of fact, preferably 
by citing the paper in which it appeared” (p. 81). Citation is therefore an important discourse-level skill that serves 
both conceptual and social functions; citations can build relationships among claims, evidence, and ideas, and they 
can also signal belief in the value of an author’s work (as evidenced by use of citation-related metrics in academic 
promotion and tenure decisions). Citation can help claims of knowledge enter a disciplinary discourse, if conclu¬ 
sions or evidence is restated in certain, unqualified terms. Horn (2001) suggested that when writers remove hedges 
or qualifications from statements cited from other documents, they help that information to be gradually converted 
from tentative knowledge to knowledge that is generally accepted within a disciplinary community. However, scholars 
have a tendency to maintain the uncertainties present in original research when they cite it; evidence from linguistic 
analyses shows that writers retain hedges and qualifications made by the original authors over 60% of the time (Horn, 
2001 ). 

It is likely that students beginning to develop their inquiry skills will be less attentive to the nuances of representing the 
certainty of cited statements and will be likely to err on the side of interpreting and representing knowledge claims gathered 
from sources as certain, rather than tentative. As noted by Goldman and Scardamalia (2013), authors are sometimes 
explicit about why they are choosing to include or cite others’ work in their own writing (Goldman & Rakestraw, 2000), 
but when texts lack explicit cues to the author’s evaluations of other sources or findings, readers tend to assume that 
the author believes in the cited source’s conclusion and evaluates it positively, consistent with Gricean maxims (Grice, 
1975). Thus, the presence or absence of hedges, qualifications, and evaluations of embedded sources can influence readers’ 
interpretation of and belief in the knowledge claims being put forward in ways that might not be explicitly evident to them 
(i.e., people have a tendency to assume that information is credible unless given indications otherwise, and so they might 
not notice that such evaluative or qualifying comments are absent). Interestingly, even at the highest levels of professional 
research practice, writers receive little guidance on how to structure citations or excerpts from other sources. As Horn 
observed, “[I]t is unfortunate that references on how to write a research article, such as style guides ... or textbooks 
. . . usually describe only different format styles for a citation. References do not explain how the citation should be 
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worded in order to relate the older research to the current work” (p. 1088). Despite these challenges, it is important 
for students to develop citation skills and, at a more conceptual level, to develop an understanding of how and why one 
should go about incorporating information from other sources into one’s own projects, which is arguably a more critical 
competency for successfully engaging in a research enterprise than being able to simply provide a list of well-formatted 
references. 

Difficulties in Writing From Multiple Sources 

Evidence suggests that students have difficulty with genres of writing that include citing and integrating sources, such as 
annotated bibliographies and literature reviews, which are commonly required assignments in college courses (Burstein, 
Elliot, & Molloy, in press). Understanding the elements of these assignments that pose difficulty for students can give some 
indication of the types of skills and prerequisites that might be addressed in K- 12 education, as emphasis shifts toward 
an increasing focus on engaging students in research from multiple sources (Coiro & Kennedy, 2011). 

Synthesizing Information From Sources 

Froese, Gantz, and Henry (1998) summarized issues that early undergraduates often experience when writing literature 
reviews, which require them to “analyze, evaluate, and creatively synthesize” the published work of others. Common 
difficulties that students encounter when constructing literature reviews in the social sciences include conceptual inade¬ 
quacies, such as failing to examine or evaluate the details of a study’s methodology or the quality of the evidence provided 
when reading and extracting information from texts; problems with inferring relationships among variables examined in 
other studies; misuse of sources, including a lack of knowledge of proper citation practices in the discipline and confu¬ 
sion between primary and secondary sources (see Froese, Boswell, Garcia, Koehn, & Nelson, 1995); difficulty critiquing or 
evaluating articles in literature reviews and a lack of certainty about whether to include their own opinions in their writing; 
problems with making relevant comparisons between articles, such as focusing on differences or comparisons between 
studies that were irrelevant or unhelpful for addressing their research question; failure to appropriately integrate infor¬ 
mation across sources by drawing intertextual connections — student’s literature reviews tend to consist of “knowledge 
telling” (cf. Scardamalia & Bereiter, 1986, p. 792) in which they summarize articles in sequence rather than integrating 
them. Issues such as knowledge telling and failure to appropriately synthesize in one’s writing are evident in the essays of 
middle school students as well, with a small proportion of students including synthesizing inferences in their responses 
to multiple-text inquiry tasks (Goldman et al., 2011). 

Avoiding Plagiarism 

Developing students’ writing is also more likely to be hindered by language issues with respect to both production and 
comprehension. For example, middle school students tend to look for sources that are comprehensible to them, making 
it easier to translate or restate information from those texts into their own words (Farge & Beheshti, 2000). Despite this 
criterion, which seems in some ways mindful of the idea that one must paraphrase or otherwise transform information 
drawn from sources before incorporating them into one’s own work, evidence suggests that sixth-grade students believe 
that by simply retyping the text, rather than cutting and pasting, one avoids committing plagiarism. This reflects an appar¬ 
ent misunderstanding of the concept of plagiarism, which is constrained in part by students’ limited vocabulary and topic 
knowledge. For example, Farge & Beheshti (2000) found that some students were more likely to plagiarize from books 
because “the books use all the good words” (p. 1073), whereas they felt it somewhat easier to translate the more casual 
language present in some Web texts — particularly those designed for children — into their own words. Transformation 
of information from multiple sources into students’ own language reflects greater conceptual understanding and is a goal 
of multiple-source integration tasks (Wiley & Voss, 1999). 

Clearly, multiple-source comprehension and integration, which requires proficiency with a constellation of reading 
and writing skills, poses a challenge for students at varying developmental levels. In sum, the multifaceted process of 
inquiry poses a challenge for many students, as they must demonstrate proficiency with many different skills in order to 
successfully conduct an extended investigation. In the next section, we discuss the role of metacognition in coordinating 
multiple aspects of extended inquiry. 
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Strategic and Metacognitive Influences on Inquiry Skills 

Coordinating and strategically deploying appropriate inquiry skills also requires metacognitive and self-regulation skills 
(Azevedo & Cromley, 2004; Edelson, 2002; White & Frederiksen, 2005). Metacognition can be defined as “one’s knowledge 
concerning one’s own cognitive processes or products or anything related to them” (Flavell, 1976, p. 232) and includes 
both knowledge and regulation of one’s own thinking. The importance of metacognition in inquiry has been heavily 
underscored in approaches to inquiry learning, from the perspective of science education and the learning sciences. 

Indeed, mounting empirical evidence shows that individual differences in metacognitive ability (e.g., Coiro & Dobler, 
2007; Graesser et al., 2007; Griffin, Wiley, & Salas, 2013; Stadtler & Bromme, 2007, 2008), prior topic knowledge (e.g., 
Braten, Stromso, & Salmeron, 2011), or epistemic beliefs 1 (i.e., beliefs about the nature of knowledge and knowing; for a 
review, see Braten, Britt, et al., 2011) can all affect students’ performance on inquiry tasks. For example, epistemic beliefs 
have a demonstrable impact on students’ engagement with multiple sources in the context of inquiry tasks. Individuals 
with an absolutist epistemology believe that assertions are facts about the world that can either be correct or incorrect, 
knowledge comes from external sources, and knowledge is certain (cf. Kuhn, Cheney, & Weinstock, 2000). Those with 
an evaluativist epistemology believe that assertions are judgments that can be evaluated using argument and evidence 
and that knowledge is generated by human minds and is fundamentally uncertain (Kuhn et al., 2000). In contrast to 
those with absolutist views, for whom information can be evaluated as correct or incorrect with respect to an external 
reality, students with an evaluativist epistemology, for whom information must be evaluated through critical thinking, are 
better able to integrate information from multiple sources in the context of online inquiry tasks (Barzilai & Zohar, 2012; 
Braten, Stromso, et al., 2011). It is important to anticipate and, if possible, account for the influence of such variables 
on students’ assessment performance, particularly in summative contexts. In this section, we review research regarding 
students’ difficulties with metacognitive processes in inquiry contexts and instructional strategies for overcoming those 
challenges. 

Student Difficulties With Metacognition and Self-Regulation in Inquiry Tasks 

Metacognitive aspects of inquiry from multiple sources, such as planning, monitoring, and evaluating the results of one’s 
information-based problem solving, pose a challenge for learners of all ages, from children through adults (Walraven 
et al., 2008). Without strong metacognitive skills, students experience a number of issues with inquiry. In online contexts, 
some researchers have observed that fifth- and sixth-grade students have great difficulty monitoring the results of their 
online search processes, failing to record useful URLs or to keep a record of searches that had yielded success (Hirsh, 1999; 
Hoffman et al., 2003; Large & Beheshti, 2000). Instead, students often reconstructed or repeated their searches, which was 
particularly time-consuming (Bilal, 2001; Large & Beheshti, 2000) and unnecessary, leading to a significant amount of 
wasted time (Hoffman et al., 2003). Evidence from work in middle school science classrooms suggests that students do 
not systematically design and carry out procedures and often fail to plan or to effectively execute plans, forgetting what 
they have already accomplished (Edelson, 2002; Krajcik et al., 1998). These problems are largely metacognitive in nature; 
for example, according to Edelson (2002), students’ problems include an “inability to recognize when they need to keep 
records, failure to plan and monitor their progress effectively, and difficulty reconciling conflicting evidence” (p. 113) 
when conducting scientific inquiry from multiple documents and technological resources. 

Students’ self-regulatory behaviors also differ from those of expert inquirers. For example, a think-aloud study of dif¬ 
ferences between the information-problem solving processes of first-year undergraduates and fifth-year graduate students 
(Brand-Gruwel et al., 2005) found that experts engaged in more frequent regulation of their information problem-solving 
processes, demonstrating more monitoring (paying attention to one’s task performance) and steering behavior (deciding 
what activities must be performed, including planning and deciding what one should do next) than novices (Brand- 
Gruwel et al., 2005). These regulation activities are associated with effective and efficient use of problem-solving strategies 
in project-based learning environments (e.g., Land & Greene, 2000). However, Brand-Gruwel et al. (2005) did not find 
direct evidence that the graduate students’ processes were more efficient per se. Interestingly, both experts and novices fre¬ 
quently used cut-and-paste features to incorporate direct excerpts from Internet texts in their written arguments, though 
experts were more likely to indicate the source of those excerpts, to make connections between the excerpts and their 
arguments, and to have overall better writing skills in terms of organization and writing style. It is likely, then, that greater 
use of self-regulatory strategies is associated with better inquiry performance. 
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Strategies to Support Metacognition in Inquiry Learning Environments 

Research on technologically enhanced inquiry learning environments have demonstrated the value of metacognitive 
scaffolding in supporting students’ inquiry learning in science domains (e.g., Eslinger, White, Frederiksen, & Brobst, 
2008; Graesser et al., 2007; Linn, Clark, & Slotta, 2003; Quintana et al., 2004; White & Frederiksen, 1998; Wiley et al., 
2009; Zhang & Quintana, 2012). In a review of inquiry learning environments, de Jong (2006) concluded that the most 
effective of these environments includes tools that scaffold the learning process; prespecified questions or hypotheses 
and background information on the topic; supports to help students plan and monitor their inquiry process, such as a 
sequence of assignments; or hints for completing open-ended task components efficiently. As an example, prompts that 
suggest particular experimentation strategies (e.g., try investigating only one variable at a time; Kuhn & Pease, 2008, 
p. 523) have been demonstrated to be useful in developing students’ scientific experimentation skills — in particular, 
their learning of the control-for-variables strategy, which is critical in making appropriate causal inferences. Students 
often fail to apply this strategy without prompting, leading to inefficient and ineffective hypothesis-testing procedures 
(Kuhn & Pease, 2008). 

Using Scaffolding to Automate Low-Level Tasks 

Building in scaffolds that support students in the metacognitive elements of inquiry, such as planning, monitoring, and 
regulating their inquiry processes, can help to free up students’ cognitive resources to engage more deeply with the tasks. 
Thus, in an assessment context, such supports can also provide more precise measurement of students’ skills in subtasks 
where metacognitive deficits might substantially limit students’ performance. Zhang and Quintana (2012) presented a 
recent example of using metacognitive scaffolding to support students’ science inquiry learning. Given that students have 
a limited time in which to conduct inquiry tasks, which require the coordination of different types of knowledge and strate¬ 
gies at different phases, the authors suggested that time wasted with low-level activities detracts from the time devoted to 
higher order cognitive activities, such as reading and analyzing a set of resources that meet criteria for relevance and reli¬ 
ability. They developed and tested a technology-enhanced notebook tool that automated low-level but memory-intensive 
tasks (e.g., automatic URL bookmarking and recording of all search histories) but gave students prompts to read, evaluate, 
and engage strategically with text content. In an extended inquiry task with sixth-grade students, use of the digital note¬ 
book was associated with deeper engagement, a greater focus on high-level cognitive activities, more on-task behavior, and 
better planning and monitoring of inquiry progress compared to a comparison group that conducted online inquiry using 
a paper notebook to record and monitor their progress. This study suggests that metacognitive scaffolding can support 
students in conducting deeper inquiries. 

Using Scaffolding to Build Understanding of Integrated Inquiry Process 

Scaffolding can also support students in understanding the purpose for engaging in the various aspects of inquiry, which 
is important to the extent that students’ understanding of the task they are engaging in constrains and motivates their 
interactions and engagement with multiple sources (cf. Rouet & Britt, 2011). When instructors make explicit connections 
between component tasks (e.g., evaluating the credibility of a Web source) and the ultimate project goals (e.g., producing 
an informative and reliable brochure about healthy eating), middle school students are better able to apply the critical 
reading skills they learned while constructing their final brochure (Kuiper et al., 2009). This finding suggests that while 
students struggle with some aspects of inquiry skills, quality instruction (in particular, instruction focused on getting 
students to understand the relevance of particular inquiry activities for achieving one’s purposes or goals) can improve 
their understanding of the nature of inquiry and thus enhance performance. 

Considering Individual Differences in Scaffolding Effectiveness 

Providing support is likely to be even more critical for younger or less advanced students, who are more likely to have 
trouble in inquiry environments that provide minimal guidance or are too open-ended (e.g., Klahr & Nigam, 2004). Qual¬ 
ifying this, though, is research in science education indicating that individual differences such as prior topic knowledge 
and metacognitive skill influence the scaffolding’s effectiveness, such that some learners are able to benefit more than 
others from certain types of scaffolding (e.g., Bulu & Pedersen, 2012; H.-S. Lee & Songer, 2004). A study of over 300 
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sixth-grade students using a scaffolded hypermedia learning environment found that students with high knowledge and 
metacognitive skill (i.e., students with ample mental resources to engage with task content and to monitor and regulate 
their own thinking) appear to benefit little from scaffolding, whereas students with low knowledge or metacognitive skill 
do seem to benefit from the use of scaffolds during complex problem-solving tasks (Bulu & Pedersen, 2012), though some 
scaffolds might be more effective for knowledgeable students (H.-S. Lee & Songer, 2004). The use of scaffolding, then, must 
be designed carefully with respect to the characteristics of the population that is intended for a particular instructional 
unit or assessment. 

Knowing Why and When To Use Particular Strategies 

It is important to note that merely providing metacognitive scaffolds to support inquiry does not ensure that students can 
take advantage of them. For example, Edelson (2002) described a design-based research project that introduced a tool 
called the Progress Portfolio in middle school classrooms to support the development of students’ reflective inquiry skills, 
emphasizing metacognitive elements of inquiry. The Progress Portfolio software consisted of a data camera for capturing 
images of the computer screen as well as a portfolio for storing, organizing, and annotating those images. After introducing 
this tool, Edelson (2002) 

continued to observe that students either lack or fail to apply inquiry strategies that would allow them to take 
advantage of the storage, annotation, and organizational tools we provided. For example, they did not necessarily 
employ effective strategies for deciding what to record or they failed to look for previously stored resources when they 
might have helped them. (p. Ill) 

Thus, simply learning about particular strategies is insufficient to ensure the development of critical thinking and scientific 
reasoning skills. Students must also develop what Kuhn calls metastrategic competence — the ability to reflect on one’s 
knowledge of specific strategies, to organize and manage this knowledge, and to deploy strategies according to one’s goals 
(Kuhn & Dean, 2005; Kuhn & Pease, 2008). That is, students need to develop not only strategies for engaging in inquiry but 
also the knowledge of why and in what circumstances particular strategies might be appropriate. Understanding why and 
when to use particular strategies increases the likelihood that a strategy will be deployed appropriately to solve problems; 
therefore, scaffolding is more likely to be effective to the extent that it can support the development of both strategic 
and metastrategic knowledge and skills. This kind of scaffolding would be critical for any inquiry learning environment 
intended to provide lasting knowledge of various inquiry strategies and the way they are deployed in a coordinated fashion 
to achieve the central goals of inquiry and information gathering; analysis, evaluation, and synthesis; and communication 
and presentation of results. 

Summary 

We presented a review of literature relevant to developing a framework of research and inquiry as a key practice in ELA. 
Consideration of these issues is important in developing designs for assessment tasks intended to provide evidence of stu¬ 
dents’ inquiry skills; therefore, the work reviewed can inform the development of designs for scenario-based assessments 
that require students to perform a series of progressively more challenging research activities. The use of scaffolds in a 
scenario-based assessment can also provide more information about test takers, potentially allowing more precision in 
estimating students’ proficiency with particular inquiry skills, which can support targeted instruction of skills in which 
students have specific difficulties. In the next section, we describe a set of hypothesized learning progressions for research 
and inquiry that are intended to support the design of such assessments. 

Research and Inquiry Learning Progressions 

The preceding analysis of research and inquiry as a key practice has resulted in a set of hypothesized developmental pro¬ 
gressions, or learning progressions (Heritage, 2008) representing the skills needed to conduct research, with a particular 
focus on inquiry involving multiple sources (cf. Goldman et al., 2011; Rouet & Britt, 2011). In this section of the paper, 
we present learning progressions for a number of skills identified as part of our analysis of the key practice conducting 
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research and inquiry, as described previously. First, we briefly describe the structure of the CBAL ELA competency model 
(Deane et al., 2015; O’Reilly & Sheehan, 2009; Sabatini et al., 2013). Next, we present definitions of nine key skills associ¬ 
ated with the practice of research and inquiry and describe how they are organized with respect to the ELA competency 
model as well as the phases of inquiry defined in Figure 1. Finally, we present detailed learning progressions associated 
with each of the nine skills; these learning progressions are presented in tables to provide precise specifications of how 
student skills might be expected to develop over time. This level of specification is intended to support both assessment 
development and instruction. 


The CBAL English Language Arts Competency Model 

The CBAL ELA competency model (Deane et al., 2013; Deane et al., 2015; O’Reilly & Sheehan, 2009; Sabatini et al., 2013) 
provides several lenses through which to analyze literacy practices, including five modes of cognitive representations and 
three types of cognitive processes. We apply each of these dimensions in turn to the analysis of the key practice, conducting 
research and inquiry. 

Modes of Cognitive Representation 

The CBAL ELA competency model suggests that students need to be able to represent ideas in the social, conceptual, 
discourse, verbal, and print modes. Each of these types of mental representations is defined briefly. 

• Social mode refers to representations of people and their communicative purposes and requires students to think 
of reading and writing as social actions. This mode involves the understanding that all sources and authors have 
a particular point of view and that this perspective shapes how they understand or present information. Social 
representations depend heavily on an underlying theory of mind or social cognitive capacity. 

• Conceptual mode involves the development of knowledge about the world in the form of mental models of concepts 
and phenomena. Conceptual reasoning involves access to prior knowledge, which is associated with comprehen¬ 
sion, inference, logical reasoning, and argumentation. Conceptual representations also involve decisions about how 
to enrich one’s existing knowledge through inquiry and decisions about how to represent information meaning¬ 
fully. Conducting research and inquiry involves mainly a combination of social and conceptual representations 
(i.e., relations among ideas and the sources of those ideas). 

• Discourse mode involves attention to the purposes and forms of texts and other documents. Here, the represen¬ 
tations focus on text structure and function, such as the purpose of headings, indices, and other organization 
functions; ability to make inferences when navigating texts with hyperlinks; and appropriate methods of citing and 
incorporating information from sources into one’s writing. Students’ comparison of the structures and functions of 
multiple documents (e.g., analyses of document type or format) also require discourse-level representations. There¬ 
fore, discourse representations are also important for locating and comparatively evaluating information during 
research. 

• Verbal mode represents the structure and meaning communicated by language; in the context of research and 
inquiry, this mode concerns the development and use of specialized vocabulary. These skills are in focus in another 
key practice, building and sharing knowledge, which emphasizes skills in learning text content and building vocab¬ 
ulary knowledge that are considered prerequisite for engagement in research and inquiry (O’Reilly et al., 2015). 
Therefore, we do not address the verbal mode in detail. 

• Print mode involves representations of spelling and writing conventions. Mastery of print conventions often occurs 
in the early grades, but extended writing as required for the communication phase of research and inquiry neces¬ 
sarily builds on fluency with decoding and transcription skills. For a detailed analysis of the print mode, see Feng, 
Sabatini, Deane, Sands, and Foley (2015). 

According to this analysis, the key practice, conducting research and inquiry, primarily requires knowledge and skills 
in social, conceptual, and discourse modes. For each of the nine research and inquiry skills, we identify the mode of 
representation that is most pertinent; these classifications are presented as a part of the skill definitions presented in a 
later section. 
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Table 1 Skill Definitions for the Key Practice Conducting Research and Inquiry 


Phase 

Skill 

Definition 

Mode 

Inquiry and information 
gathering 

Asking guiding questions 

Pose and investigate focused questions to explore and 
build consolidated knowledge of a topic and use 
this knowledge to answer inquiry questions. 

Conceptual 


Testing hypotheses 

Engage in hypothesis-testing procedures to evaluate 
the degree to which data supports one hypothesis 
over another and explain connections between 
results and causal explanations for phenomena. 

Conceptual 


Locating sources 

Set goals and plans for locating useful information 
resources, differentiate among types of resources, 
and execute appropriate search strategies to obtain 
relevant information. 

Discourse 

Analysis, evaluation, and 
synthesis 

Reconciling perspectives 

Identify authors’ perspectives, motivations, and 
biases; use knowledge of authors’ perspectives to 
interpret and evaluate text content; and reconcile 
disagreements across texts by appealing to 
differences in perspectives. 

Social 


Evaluating sources 

Attend to and critically evaluate information as a 
function of source characteristics, including task 
relevance, reliability of content, credibility and 
expertise of authors, and usefulness for one’s task. 

Conceptual 


Integrating multiple 
formats 

Recognize common information presented in 
different modalities and formats; recode 
information in different modalities; produce 
discussions that draw on multiple forms and 
formats of evidence. 

Conceptual 


Comparing, contrasting, 
and organizing 

Compare and contrast the discourse structure and 
content of multiple sources on a topic and organize 
information with respect to salient categories or 
goal-driven purposes. 

Discourse 

Communicating and 

Synthesizing research 

Integrate and embed descriptions and evaluations of 

Conceptual 

presenting results 

results 

information drawn from multiple sources into a 
coherent synthesis to communicate the results of 
one’s inquiries to others and contribute to a 
developing body of knowledge about a topic. 



Citing and using sources 

Understand and explain the use of sources to support 
claims; use sources selectively to support one’s own 
purposes; and incorporate information drawn 
from sources in written texts, using summary, 
paraphrase, and quotation as appropriate. 

Discourse 


Note. Phase = phase of key practice; mode = Mode of representation from English language arts (ELA) competency model. 


Modes of Cognitive Processing 

The ELA competency model also distinguishes among three modes of cognitive processing, namely interpretive, expres¬ 
sive, and deliberative processes. Interpretive processes correspond to fluent reading, text comprehension, and evaluation 
of texts and documents. Expressive processes correspond to fluent written composition, spoken production, or use of other 
methods to communicate meaning to others. Finally, deliberative processes involve reflective and metacognitive processes 
that support critical evaluation and strategic decision making. Thus, each of the nine inquiry skills we identify as impor¬ 
tant to the key practice can be decomposed in terms of its application to reading, writing, and critical thinking processes. 
This analysis forms a critical part of the organization of the learning progression tables. Though these three cognitive pro¬ 
cesses are often coordinated in instruction, and in advanced practice, it is important to specify how each skill intersects 
with the various types of processes, particularly for the purpose of assessment and diagnosis of student difficulties (i.e., are 
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Figure 2 Mapping from the key practice conducting research and inquiry, to assessment structure, to learning progression level 
descriptors for the CBAL Invasive Plant Species summative form. 


difficulties with citing sources due to trouble with writing skills or with students’ ability to read and interpret the text). In 
the next section, we define these skills and outline a structure for the learning progressions, connected to the competency 
model dimensions described here. 


Skills Required for Conducting Research and Inquiry 

At the beginning of this paper, we identified three phases of the key practice, conducting research and inquiry: inquiry 
and information gathering; analysis, evaluation, and synthesis of multiple sources; and communication and presentation 
of results. On the basis of the literature reviewed previously, we present a set of nine skills that are critical for proficiency 
with research and inquiry in ELA. Specifically, we have postulated learning progressions for the following skills: asking 
guiding questions; testing hypotheses; locating sources; reconciling perspectives; evaluating sources; integrating multiple 
formats; comparing, contrasting, and organizing; synthesizing research results; and citing and using sources. 

In Table 1, we present a definition of each of these skills, organized within the context of the three phases of the key 
practice (see Figure 1). In the rightmost column, we also include the primary mode of representation that performance 
of the skill involves (i.e., discourse, conceptual, or social). Several of these skills and associated learning progressions are 
revised from the previous version of the CBAL ELA competency model (O’Reilly & Sheehan, 2009; Sabatini et al., 2013). 
In addition, we propose two new skills and hypothesized learning progressions, locating sources and evaluating sources, 
which represent two critical elements of inquiry in online environments (e.g., Bilal, 2001, Braasch et al., 2009; Wallace 
et al., 2000) that were not well represented the previous ELA competency model. The learning progressions associated 
with each inquiry skill are presented next. 

Hypothesized Learning Progressions 

In CBAL research, a learning progression is defined as a description of qualitative change in a student’s level of sophisti¬ 
cation for a key concept, process, strategy, practice, or habit of mind. Change in student standing on such a progression 
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procedures or protocols, related literature) to allow another was done) documents both what investigations may 

and determining person to replicate one did and what not be quite 

whether the results the experiment happened as a result appropriate tests of 

confirm or disconfirm the hypotheses 

the hypothesis 
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What are the consequences of my ideas? Do they fit the facts? Are there other ways to conceptualize my subject? 
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may be due to a variety of factors, including maturation and instruction. Each progression is presumed to be modal—to 
hold for most, but not all, students. Finally, it is provisional, subject to empirical verification and theoretical challenge 
(Deane, Sabatini, & O’Reilly, 2011). Some of the descriptions are grounded strongly in empirical literature, while others 
are extrapolated from available evidence when research is thin. All descriptions are considered to be hypotheses that are 
open to empirical verification or falsification. 

To organize the qualitative changes along a developmental continuum, we label the hypothesized learning progressions 
for research and inquiry with respect to five levels, namely, preliminary, foundational, basic, intermediate, and advanced. 
These labels are used relative to a scale of how well students are able to conduct research in the interpretive, expressive, 
and deliberative modes. For example, a student at the preliminary level is learning the beginning moves that are required 
for inquiry but uses relatively simple methods for evaluating texts and sources and has a limited capacity to integrate 
information from multiple texts. At the basic level, students can draw conclusions from multiple sources that incorporate 
information about sources’ reliability and other characteristics but may only present one-sided information when con¬ 
structing their own arguments. A student at the advanced level conducts inquiry with relative fluency, using proficient 
metacognitive and self-regulation skills to develop integrated knowledge and construct extended analyses of important 
problems and issues within specific fields of study. 

These levels describe a modal pattern of development in students’ knowledge, skills, and abilities in the practice of 
research and inquiry. Not all students will go through all the levels in the same order or reach the advanced level at a 
certain grade. However, for the purposes of developing ETA assessments, it is necessary to specify the range of grade 
levels for which certain competencies may be relevant, acknowledging that instruction in inquiry often does not begin 
until middle school and that the research reviewed above suggests many deficits in even undergraduates’ research skills. 
We propose the following alignment as a general guideline: 

• Preliminary: Kindergarten to third grade (early elementary) 

• Foundational: Fourth to sixth grade (upper elementary) 

• Basic: Seventh to ninth grades (middle school) 

• Intermediate: 10th to 12th grades (high school) 

• Advanced: Undergraduate (college ready and beyond) 

In Table 2, we present an overview of the progressions for each skill in the key practice model in parallel. This table 
includes descriptions of key knowledge, skills, and abilities to illustrate how inquiry skills may develop in concert from 
preliminary to advanced (i.e., collapsing across these processing modes). Tables 3 through 11 present the hypothesized 
learning progressions for each of the nine skills, presented as the intersections among the five levels of development 
(from preliminary to advanced) and the varieties of reading-focused (interpretive), writing-focused (expressive), 
and strategic or metacognitive (deliberative) processes. Thus, the basic structure of each progression is a 3 X 5 table, 
describing the primary aspects of proficiency with the skill for each level and each mode of thought (e.g., evaluating 
sources^preliminary^interpretive). Within each of these 15 cells, we differentiate between an achievement, a statement 
of what students are expected to achieve at this level, and a description of the critical limitations of students’ skill (with 
the exception of the advanced level, where it is assumed there are no limitations on students’ performance of inquiry 
tasks). 

As noted above, the learning progressions presented in the following tables are provisional and subject to empirical 
validation and revision. To the extent that these learning progressions reflect the scope of the domain and a normative 
development pattern, this competency model can support the development of assessments of research and inquiry skills. 
The progressions might also yield suggestions for particular instructional moves or interventions that might be used to 
help students progress to the next level of skill proficiency (Deane, 2011; Deane, Sabatini, et al., 2011). 

Mapping from Key Practice to Assessment Design 

Specifying a model of the key practice, conducting research and inquiry, as defined previously affords a range of possi¬ 
bilities for task types and sequences that might be incorporated into a scenario-based assessment of research and inquiry 
skills. The flexibility of the model defined here is useful, given the complex and multifaceted nature of inquiry and the 
need for assessment developers to make principled decisions regarding the skills that will be measured in the assessment 
and the tasks and situations that provide evidence of those skills. In this section, we provide an illustration of how the 
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Invasive Plant 

Task'fA J f 1°™ 


Question » Timer 

r 


J 





/T 


rtj 




•People in lots of different places are worried about the 
damage caused by invasive plant species. 

•There are invasive plant species in our local gardens and 
state parks. 

•Some invasive plant species clog creeks. Then, during heavy 
rains, the water spills over the banks and causes flooding. 

•Kudzu is one of the worst invasive plants. It came from 
Asia and is now a big problem in lots of Southern states. 

•Students can volunteer to help park rangers pull out 
invasive plants that have caused problems in state parks. 

• In Hawaii, people are worried that invasive plants might 
make some native Hawaiian plants disappear. 

•Because of climate change, some plants have become 
invasive. 


Directions 

Your List | 

Directions: Type five other important research questions 
in the boxes below, one question in each box. Your 
questions should apply to all invasive plant species and be 
important for people to know about. 

1 


2 


3 


4 


5 



Figure 3 Screenshot of constructed-response items in Task 1A (Asking Guiding Questions — Generate Subquestions). 


model of conducting research and inquiry as a key practice in ELA can be used to organize a sequence of assessment 
tasks and items within the context of a scenario-based assessment. The example assessment described below is intended 
to illustrate the utility of the activity diagram (Figure 1) as a framework for designing a scenario that requires a combi¬ 
nation of inquiry activities organized in a sequence that is consistent with our definition of the key practice. While the 
activity diagram could be used to support very different instantiations of the research and inquiry process depending on 
one’s goals, we provide one example of how the practice provides a coherent framework for aligning competency models, 
learning progressions, and task designs. Specifically, we illustrate how the high-level sequence of activities as defined in 
the key practice is instantiated in particular assessment goals, which are tied to specific performance descriptions from the 
learning progressions for targeted skills. Selection of measurement targets helps to define the extent to which a particular 
scenario design will generalize across contexts; that is, tasks can be selected and prioritized so that the overall assessment 
samples a range of targeted skills in a principled way. 

Figure 2 presents an analysis of one CBAF summative assessment, Invasive Plant Species, in terms of the key practice of 
research and inquiry, organizing the main assessment tasks with respect to the major phase of the key practice and aligning 
each task with a performance description from the revised learning progressions (Tables 3-11). The mapping of learning 
progression descriptions to specific item and task types is illustrated in Figures 3 - 7. In Figure 2, the assessment tasks are 
represented by green boxes, and green arrows illustrate the sequence of activities across various subgoals of conducting 
inquiry. 

In this assessment, students are presented with a scenario in which they will work with a simulated team to produce a 
brochure about invasive plant species to be distributed at an upcoming environmental science fair. Students are presented 
with some background information about the topic and then engage in a series of five tasks that provide evidence of 
inquiry skills, including generating research questions, evaluating the relevance and reliability of Web sites, organizing 
information by research question, revising part of the pamphlet written by a team member, and drafting two sections of 
the pamphlet based on information gathered and assembled in the previous tasks. As students complete the five tasks 
in this scenario, they first cycle between the phases of inquiry and information gathering and analysis, evaluation, and 
synthesis (Tasks 1A, IB, and 2) before moving on to the phase of communicating and presenting results in the final two 
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Question 0 Timer 


Invasive Plant 

«gB ) r ^ J 



^ il | "Wl l£lnQ. 


\ t) 


Topic 


l _l 




Search-a-rama 



Environmental Research Center. December 2003 
Scientists discuss the possibility of investigating plants introduced from 
other countries to see whether they might pose a problem for various 
ecosystems native to the United States. 

Be Green. Invasives’’ TV episode. January 2007 

Take a tour of different environments with a scientist to learn about the 

importance of controlling invasive non-native plants, insects, and 

animals. 

Suzanne's GardenBloa. June 2007 

I have been sharing a list of plants with friends that—based on my 
experience in the garden—seem safe to grow because they are not 
too invasive. But no guarantees! These are just my personal 
favorites, and they are really pretty! 

U-S- Department of the Interior. Bureau of Land Management, March 

2008 

Debate about the pros and cons of water hyacinth, which can clean 
water but also seriously clog local water ways because of runaway 
growth... Originally from South America, the water hyacinth... 

National Geographic, June 2009 

Scientists have discovered several previously unknown species in an 
unexplored Mozambique mountain range. 

Invasives in YOUR house?. March. 2001 

Let our team of experts use state of the art methods for getting rid of v 


»L 


Suzanne's GardenBlog. June 2007 
I have been sharing a list of plants with friends that—based 
on my experience in the garden—seem safe to grow because 
they are not too invasive. But no guarantees! These are just 
my personal favorites, and they are really pretty! 


Is this source useful for your research? 
r Yes, because the source meets the guidelines. 
No, mainly because the source is 
not relevant 
c biased 

r not authoritative 
r outdated 


Figure 4 Screenshot of selected-response items in Task IB (Evaluating Sources). 


Invasive Plant 

Species /- 

Task 2 J 1 of 16 



Question 0 Timer 

r 


C6r\ I' 


1 Vnitlxjq. 




Note Card 


www.ucsusa.ora. Union of Concerned Scientists Web site. April 2009 

Two major ways invasive plants come to U.S.: 

1. Ships and airplanes bring plants and seeds from 

other countries, (accidental) 

•For example, seeds may be in the water or 
ballast that is put into a ship in Brazil and 
then released into our lakes or rivers. 

2. People import exotic plants to sell, (intentional) 

•For example, in the 1800s, people imported 
purple loosestrife because it looked pretty 
and they thought it might improve their 
health. 


Read the information on the note card. Then click on the 
research question that you think it best answers. 


Mouse-over text 

ballast = rocks or other heavy material adding weight and 
stability to cargo ships 
exotic = foreign, unusual 
intentional = done on purpose 


r What is the definition of invasive species? 


How are invasive plant species introduced? 


c What kinds of environmental and economic problems can 
invasive plant species cause? 


r What are the best ways to prevent and/or control 
invasive plant species? 


Can invasive plant species ever be benef icial? 


Figure 5 Screenshot of selected-response item in Task 2 (Asking Guiding Questions — Organize Information by Subquestion). 
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Invasive Plant 
Species 
Task 3 


Question 0 Timer 


y 


2 of 2 


,r 


J 






What YOU can do! 

Since invasive plant species can cause widespread damage, 
which is really bad, it is important that kids, adults, and 
everyone join in the fight against them. Here are simple steps 
you can take to help prevent the spread of these non-native 
pests and become part of the solution. 

GO NATIVE: If you have a garden, avoid planting invasive 
species. You can learn which plants are invasive and which 
are native on the Web or at your local nursery. Some people 
buy invasive plants because their fragrance or color can be 
very nice. Some are purple. You can also work with other 
people to join organizations to help get rid of invasive 
species. 

Buy only native plants, and you'll have a great garden. That's 
one thing you can do. 

GET RID OF HITCHHIKERS: Invasive species can create 
problems. After hiking, clean your boots and backpack. 

Seeds and berries can easily attach themselves and be 
transported. A lot of them come from products that are 
shipped to the U.S, from other countries. 

TAKE ACTION: Learn to identify invasive plant species in 
your area and then educate your friends and family about the 
harm invasive plant species can cause. You might want to 
contact other people and that's fine, too. Volunteer at nature 
reserves and parks to help stop the spread of invasive plant 
species. 




Directions 

Your Answer 

Tips 


Directions: Revise the student's draft. When you 
finish, all of your text should fit inside the box 
below, without scrolling. 


What YOU can dol 

Since invasive plant species can cause widespread 
damage, which is really bad, it is important that kids, 
adults, and everyone join in the fight against them. Here 
are simple steps you can take to help prevent the spread 
of these non-native pests and become part of the solution. 

GO NATIVE: If you have a garden, avoid planting 
invasive species. You can learn which plants are invasive 
and which are native on the Web or at your local nursery. 
Some people buy invasive plants because their fragrance 
or color can be very nice. Some are purple. You can also 
work with other people to join organizations to help get rid 
of invasive species. 

Buy only native plants, and you'll have a great garden. 
That's one thing you can do. 

GET RID OF HITCHHIKERS: Invasive species can create 
problems. After hiking, clean your boots and backpack. 
Seeds and berries can easily attach themselves and be 
transported- A lot of them come from products that are 
shipped to the U.S, from other countries. 

TAKE ACTION: Learn to identify invasive plant species in 


Figure 6 Screenshot of constructed-response item in Task 3 (Revision — Draft, Revise, Edit, and Publish Texts). 


tasks (Tasks 3 and 4). 2 This design represents a relatively straightforward mapping onto the major activities of the key 
practice, in that each assessment task corresponds to only one targeted skill, but more complex designs are possible. The 
scenario measures learning progression levels ranging from preliminary to basic proficiency, which is an appropriate range 
of difficulty for middle school students (Deane, Fowles, et al., 2011). Taken together, this example serves to illustrate one 
way in which the key practice and associated learning progressions can be instantiated in a scenario-based assessment 
task that affords measurement of distinct, but complementary skills, organized in a way that is consistent with real-world 
contexts in which those skills might be applied (i.e., in the context of the practice of conducting inquiry). 


Conclusion 

The CBAL ELA competency model is organized around 11 key practices that take the form of activity systems derived from 
sociocognitive theory as well as theory in literacy and the learning sciences more generally. Each key practice is described 
in terms of a coordinated set of activities or phases. This paper defines and elaborates one of those key practices, conduct¬ 
ing research and inquiry, decomposing it into three phases. The first phase, inquiry and information gathering, concerns 
activities related to developing and planning investigations, including understanding the task, defining research ques¬ 
tions, and identifying sources of information or data collection methods that may yield answers to those questions. The 
second phase, analyzing, synthesizing, and evaluating, requires a collection of cognitive processes related to close read¬ 
ing, comparison and contrast, and integration of information presented in multiple distinct sources, which may include 
information that varies in terms of author, credibility, presentation format, complexity, and so on; the ability to skillfully 
build meaning from multiple sources while taking into account their reliability and credibility is the focus of this phase. 
The third phase, communication and presentation of results, involves skills related to citing and writing from sources, 
including appropriate use of bibliographic techniques, and effective incorporation of others’ texts into one’s own writ¬ 
ten products. Together, these three phases form the basis of our model of engagement in the key practice of conducting 
research and inquiry in ELA. 
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Timer 


CHA 




Notes ~~| Tips "1 Section 1 'I section d 


Directions Section 2 Section 3 J 

1. What is the def inition of invasive species? 

•Native species - are they ever defined as 
invasive? Several sources say yes, they can 
become invasive, if there has been a big 
change in their environment. 

Example: With global warming, some native 
plants/animals have spread to areas that 
used to be too cold for them to thrive. Now 
they're hard to control. 

* 

Directions: Write the text for Section 2 of the pamphlet. You may 
revise the heading. 



[student's heading goes here] 


•Not all foreign species harm the 
environment. Scientists define “invasive 
species" as “a species that is non-native to 
an ecosystem and whose introduction can 
harm the environment, the economy, or even 
human health." 





• Why are so many nonnative plants 

considered invasive? Because when they 
come into a new environment, they might not 
have their usual natural enemies or diseases. 
With nothing to stop them, they can 





overwhelm or kill off native plants. 

V 





Figure 7 Screenshot of constructed-response item in Task 4 (Synthesizing Research Results). 


Each phase is further specified as a set of learning progressions that correspond to key skills associated with the practice. 
The purpose of the progressions is to characterize qualitative shifts in students’ abilities as they develop from novice 
to more expert levels, on the basis of current theory and empirical work related to those skills. The progressions can 
be used to guide assessment development and to support teachers in making evidence-based instructional decisions. 
Here, we presented detailed specifications for learning progressions, and in the final section of the paper, we illustrated 
how these learning progressions might be organized with respect to the phases of the key practice in the context of an 
example assessment design for measuring middle school inquiry skills in ELA. This work represents an important step in 
supporting the assessment of students’ skill in building knowledge by synthesizing multiple sources. 

Further work is needed in order to validate the hypothesized learning progressions presented here in terms of their 
ability to accurately capture and describe the developmental sequence by which research and inquiry skills develop; we 
are currently developing a set of items aligned to various levels of the learning progression for evaluating sources in order 
to test specific hypotheses about how these skills develop and the extent to which there are dependency relationships 
among certain aspects of source evaluation (i.e., relevance and reliability judgment). In addition, research that evaluates 
the instructional utility of the key practice, associated learning progressions, and scenario-based tasks must be conducted 
in order to determine the extent to which practicing teachers can make use of the learning progressions and assessment 
results to improve instruction, and to what extent they need support in doing so. This work will contribute to our under¬ 
standing of how research and inquiry skills develop over time, and how these critical 21st century skills can be supported 
with appropriate instruction. 

Notes 

1 These student-level characteristics are consistent with O’Reilly and Sabatini’s (2013) conception of performance moderators , 
individual differences that might affect performance on an assessment but which are considered separate from the construct 
being measured. 

2 Notably, though, the pamphlet revision task (Task 3) measures students’ skill in revision, which is part of the key practice of draft, 
revise, edit, and publish texts, rather than a skill that is specific to inquiry. In terms of the conducting research and inquiry key 
practice, revision of others’ work would fall under the subgoal of communicating and presenting results, so a link to the skill of 
revision is represented under this phase of inquiry with a gray box in Figure 2. 
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